Weblog entry #1 for Utumno
See all weblog entries made by Utumno.
#1
Question about network setup
Posted by Utumno on Thu 2 Mar 2006 at 14:53
I've got an ADSL line and an ADSL router with a 4-port switch. I connected my server ( Etch running a small forum + blog + SSH server ) to one port.
I noticed a curious thing: whenever I 'apt-get update && apt-get upgrade' it, the connection speed ( from a nearby university mirror ) almost always tops at exactly 120kB/s or a little below that.
Yesterday I brought my company's laptop ( on which I have also installed Sid ) and plugged it to another port in the switch. Then, I tried to update&&upgrade both machines at once.
Both dowloads still top at about 120kB/s.
This begs the question: if I connect 2 of my server's network interfaces to 2 ports in the switch and follow the 'Aggregating network interfaces' little HOWTO to bond them into one virtual interface, would the download speed top at 240kB/s ?
What if I get two more network cards and plug all 4 to the switch? 480kB/s ? :)
I am going to try that this weekend, but actually I've got yet another idea: it would be sweet if I could connect the two network interfaces to two ports in the switch, and NOT combine them, but just be able to tell some applications on my server to use one interface and some other to use the other.
Why would I want that? This way I'd be able to have my Apache listen on , say, ppp0 and everything else on ppp1 and be able to upgrade&&update or use p2p without hampering the performance of Apache.
Additionally, methinks this setup is also more secure: currently, the only open ports on ppp0 are 22 and 80. If I would be able to get SSHd listen on ppp1 and Apache on ppp0 I could set up iptables to block everything but 22 on ppp1 and everything but 80 on ppp0, which - from outside - should make the machine look like two machines each with just one port open, which should confuse the potential wannabe hackers a bit.
Dream? You tell me ;) ( what about routing then? I dont think I can have two default routes each one valid only for some certain application?? )
I noticed a curious thing: whenever I 'apt-get update && apt-get upgrade' it, the connection speed ( from a nearby university mirror ) almost always tops at exactly 120kB/s or a little below that.
Yesterday I brought my company's laptop ( on which I have also installed Sid ) and plugged it to another port in the switch. Then, I tried to update&&upgrade both machines at once.
Both dowloads still top at about 120kB/s.
This begs the question: if I connect 2 of my server's network interfaces to 2 ports in the switch and follow the 'Aggregating network interfaces' little HOWTO to bond them into one virtual interface, would the download speed top at 240kB/s ?
What if I get two more network cards and plug all 4 to the switch? 480kB/s ? :)
I am going to try that this weekend, but actually I've got yet another idea: it would be sweet if I could connect the two network interfaces to two ports in the switch, and NOT combine them, but just be able to tell some applications on my server to use one interface and some other to use the other.
Why would I want that? This way I'd be able to have my Apache listen on , say, ppp0 and everything else on ppp1 and be able to upgrade&&update or use p2p without hampering the performance of Apache.
Additionally, methinks this setup is also more secure: currently, the only open ports on ppp0 are 22 and 80. If I would be able to get SSHd listen on ppp1 and Apache on ppp0 I could set up iptables to block everything but 22 on ppp1 and everything but 80 on ppp0, which - from outside - should make the machine look like two machines each with just one port open, which should confuse the potential wannabe hackers a bit.
Dream? You tell me ;) ( what about routing then? I dont think I can have two default routes each one valid only for some certain application?? )
Comments on this Entry
Individual TCP connections have a performance limit based on various properties of the protocol and underlying network hardware. It is possible you are hitting this (or that the mirror is limiting traffic per connection to 120kBps - ask the admin).
In theory you may get the benefit by doing two concurrent connections in apt - a parallel "apt" tool, i.e. download one file in one connection, with another connection from the same (or different) machine downloading another. If it is per connection limits, bonding the two connections in some way will likely not improve matters much, since at 120kBps it clearly isn't simply the bandwidth on the link that is the issue.
You might also tune the TCP stack to improve this download performance, there is a lot of documentation on tuning TCP window sizes and such like on the net, do look for stuff referring to the right kernel, and TCP/IP stack implementation, as there are differences.
You'll probably do better APT wise to install a local caching proxy (Squid or Apt-Proxy), and only download the stuff once. Consider also traffic shaping, I think Wondershaper in Debian prioritises ACKs, thus improving the performance of existing connections.
In theory you may get the benefit by doing two concurrent connections in apt - a parallel "apt" tool, i.e. download one file in one connection, with another connection from the same (or different) machine downloading another. If it is per connection limits, bonding the two connections in some way will likely not improve matters much, since at 120kBps it clearly isn't simply the bandwidth on the link that is the issue.
You might also tune the TCP stack to improve this download performance, there is a lot of documentation on tuning TCP window sizes and such like on the net, do look for stuff referring to the right kernel, and TCP/IP stack implementation, as there are differences.
You'll probably do better APT wise to install a local caching proxy (Squid or Apt-Proxy), and only download the stuff once. Consider also traffic shaping, I think Wondershaper in Debian prioritises ACKs, thus improving the performance of existing connections.
[ Parent | Reply to this comment ]
Posted by Anonymous (66.190.xx.xx) on Fri 3 Mar 2006 at 16:11
The 120k download is probably controlled by the university's traffic shaping profile; they probably have limited (that kind of) traffic to 120k/sec per connection.
As for the rest, there are ways to do what you want to do, but the result will almost certainly not be what you are trying for. A better solution would be to set up traffic categorization and shaping on your router. Your router is a linux box, right? Most hardware routers have pathetic queuing and fairness characteristics; even a bare linux box is better. Wondershaper and its replacements are acceptable for balancing low latency (gaming and ssh terminal), high bandwidth (web traffic, scp or rsync-over-ssh, etc), and bulk traffic (various filesharers, traffic you don't care about, IMAP and POP, etc).
In the past I have recommended such a setup for multiple college students sharing a cable modem to stop the wars about "your download lagged me out in my game" and "you game all the time; I can't ever download" etc. It has worked quite well.
Probably the best solution, if you have the time and understanding to configure it, is Shorewall. It's a really good firewall, and now has the ability to handle shaping. Two notes: ALWAYS set your bandwidth limits 2-5% lower than you can get with a bandwidth tester like http://speakeasy.net/speedtest/ . It will speed things up significantly, by making sure that all the queue management happens on your local router rather than in the DSL head or cable provider's office. They don't care which packets go first and you do.
As for the rest, there are ways to do what you want to do, but the result will almost certainly not be what you are trying for. A better solution would be to set up traffic categorization and shaping on your router. Your router is a linux box, right? Most hardware routers have pathetic queuing and fairness characteristics; even a bare linux box is better. Wondershaper and its replacements are acceptable for balancing low latency (gaming and ssh terminal), high bandwidth (web traffic, scp or rsync-over-ssh, etc), and bulk traffic (various filesharers, traffic you don't care about, IMAP and POP, etc).
In the past I have recommended such a setup for multiple college students sharing a cable modem to stop the wars about "your download lagged me out in my game" and "you game all the time; I can't ever download" etc. It has worked quite well.
Probably the best solution, if you have the time and understanding to configure it, is Shorewall. It's a really good firewall, and now has the ability to handle shaping. Two notes: ALWAYS set your bandwidth limits 2-5% lower than you can get with a bandwidth tester like http://speakeasy.net/speedtest/ . It will speed things up significantly, by making sure that all the queue management happens on your local router rather than in the DSL head or cable provider's office. They don't care which packets go first and you do.
[ Parent | Reply to this comment ]
Posted by Utumno (211.75.xx.xx) on Fri 3 Mar 2006 at 18:16
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
[ Send Message | View Utumno's Scratchpad | View Weblogs ]
Thanks for your answers, guys!
I can see I didn't describe the situation clearly. But first, let me mention that as far as networking is concerned, I am new to anything more advanced than issuing a 'ifconfig' or a 'route' command, so please go easy on me :)
1) the router is a hardware router with the switch build in. It says 'CT 562C' on it, and there are also a whole lot of Chinese characters written all over it ( I am a software engineer currently living in Taiwan ) but those don't help me much :)
Google knows about a 'Comcast CT 562' ADSL router. Mine must be a variation of that, although it doesn't even say 'Comcast' on it, just the name of the local telco that provides the DSL line. So it is apparently a CT 562 software-modified for the telco. Besides that, I have zero knowledge about it ( this is my company's apartment, company pays for the DSL and I moved in a month ago ).
In particular, I dont even know its IP address ( shouldn't it be always the first line of a 'traceroute x.y.z.k' command? but it doesn't look like so )
2) I've done some speedtests, and they are consistent with the speed of apt-getting: always about 115-118 kB/s download, ~8 kB/s upload. So, I guess it is not the university limiting download speed per session, but the telco limiting it in its DSLAMs or in the router.
3) I've got public, but dynamic, IPs here ( I use DynDNS for the 'small server' )
4) I am looking for a way to upgrade my server's network performance. Since when I connected the laptop , both machines peaked at 120 kB/s download simultanously, I guess I can aggregate two network interfaces and (almost) double my network speed?
5) or, maybe I can hack my way into the router ( although I am probably not supposed to do that ) and disable it's rate limits ( if it is indeed the router that limits the connection speed on a single port? how to check that BTW? )
A whole lot of questions :)
I can see I didn't describe the situation clearly. But first, let me mention that as far as networking is concerned, I am new to anything more advanced than issuing a 'ifconfig' or a 'route' command, so please go easy on me :)
1) the router is a hardware router with the switch build in. It says 'CT 562C' on it, and there are also a whole lot of Chinese characters written all over it ( I am a software engineer currently living in Taiwan ) but those don't help me much :)
Google knows about a 'Comcast CT 562' ADSL router. Mine must be a variation of that, although it doesn't even say 'Comcast' on it, just the name of the local telco that provides the DSL line. So it is apparently a CT 562 software-modified for the telco. Besides that, I have zero knowledge about it ( this is my company's apartment, company pays for the DSL and I moved in a month ago ).
In particular, I dont even know its IP address ( shouldn't it be always the first line of a 'traceroute x.y.z.k' command? but it doesn't look like so )
2) I've done some speedtests, and they are consistent with the speed of apt-getting: always about 115-118 kB/s download, ~8 kB/s upload. So, I guess it is not the university limiting download speed per session, but the telco limiting it in its DSLAMs or in the router.
3) I've got public, but dynamic, IPs here ( I use DynDNS for the 'small server' )
4) I am looking for a way to upgrade my server's network performance. Since when I connected the laptop , both machines peaked at 120 kB/s download simultanously, I guess I can aggregate two network interfaces and (almost) double my network speed?
5) or, maybe I can hack my way into the router ( although I am probably not supposed to do that ) and disable it's rate limits ( if it is indeed the router that limits the connection speed on a single port? how to check that BTW? )
A whole lot of questions :)
[ Parent | Reply to this comment ]
Posted by Anonymous (196.34.xx.xx) on Sat 8 Apr 2006 at 18:44
I am looking for a linux solution that can provide the same as below without the price tag!
http://www.fatpipeinc.com/xtreme/index.html
http://www.fatpipeinc.com/xtreme/index.html
[ Parent | Reply to this comment ]