Securing OpenSSH Server [Part 1]
Posted by ahowells on Thu 10 Jan 2008 at 10:50
I'm sure if you're responsible for any server connected to the Internet with a "real" IP address, or if you port forward :22 to a box on your LAN, your logs will contain a lot of failed SSH attempts. Here we'll look at some simple solutions.
The logged entries will probably look something like the following:
sshd[22768]: Invalid user carol from 91.194.85.65 sshd[22770]: Invalid user carolr from 91.194.85.65 sshd[22772]: Invalid user sergi from 91.194.85.65 sshd[22774]: Invalid user quala from 91.194.85.65
Now these are an annoyance, but not much of a concern, as none of these usernames exist on my system - its just filling up my logs, nothing more.
There are other more alarming messages hiding in your logs too, like these:
sshd[17810]: Failed password for root from 91.194.85.65 port 44229 ssh2 sshd[17811]: Failed password for root from 91.194.85.65 port 38154 ssh2 sshd[17814]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=91.194.85.65 user=root sshd[17815]: pam_unix(ssh:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=91.194.85.65 user=root
Both of these logs come from different physical boxes, with IPs located in totally different /16's and yet it is the same IP address attacking both? Scary! Even more worrying is the fact the attacks are many hours apart, this guy has been hammering hundreds or thousands of servers all day...
What happens if they eventually get lucky? The "best case" scenario is they get an unprivileged account on your system, with which they can:
- Use you as a platform to launch further SSH attacks.
- Add your box to a "botnet" and launch DDoS attacks.
- Attempt to exploit a local vulnerability leading to privilege escalation.
- Send many thousands of spam messages through your server.
- Steal information, delete information.
If they end up with root@yourserver, things get more interesting - they can literally do anything they want with the box. You're screwed. Really.
Every single decision you ever make about these sort of issues will involve balancing usability and accessibility vs. security and safety. There are lots of options available to you, some more "annoying" to real users than others.
- 1. DenyHosts
-
Simply put this is "apt-get install denyhosts" and the default configuration is pretty good. It'll send you alert e-mails if you want them and the level at which block compromised IPs with /etc/hosts.deny is tunable.
Pros:
- User has only a few attempts to get correct password.
- Susceptability to brute force attack vector reduced lots.
- Naughty IPs can be removed after a set time, which is helpful if you've blocked yourself accidentally.
- Very simple to install on your system
Cons:
- Both fail2ban and DenyHosts had security vulnerabilities recently which could allow people to DoS your box by adding the whole world to /etc/hosts.deny
- 2. Firewall
A simple firewall rule can totally drop all traffic to your SSH server if the traffic isn't coming from your IP.
Pros:
- Stops dead most attacks, you just aren't there!
- Fairly trivial to setup. Few dependencies.
Cons:
- Impossible to maintain if your server is being accessed by lots of users from many changing IPs.
- You can't access your box easily from friends or whilst you're away on holiday from a cybercafe.
- If you bugger up when setting this up, you could break access to your server completely. :(
- 3. Strong Passwords
This is often overlooked, but a real winner. Ensure that all accounts on your system have strong passwords using cracklib.
- minimum of 8-10 characters advised
- mixed case, require at least 1-2 capitals
- numerics, require at least one number
- throw in a symbol for good measure
Generating these passwords is easy with pwgen, to make really strong passwords invoke like so: "pwgen -syB 23"
Pros:
- The simplest option
- Even a ten or twelve character strong password has trillions of possibilities. Very hard to break.
Cons:
- none, except having to remember them ;)
- 4. Public Key Authentication
Remembering all these strong passwords can be a chore. Often when you're dealing with 2-3+ boxes it's easier to just install the public part of a certificate in ~/.ssh/authorized_keys.
Then you may authenticate to the server with your private key.
Simply put: drop your key in authorized_keys and try to login by doing "ssh -i key.file user@your.hostname.here". Once you have this working you can disable other authentication methods.
Pros:
- Fairly immune to bruteforce
- Single sign-on` (same key!) on many servers without loss in security, you can use simpler (but still secure!)
- Passwords for sudo on your server to get root
Cons:
- none really, once you've mastered the basics
- keeping your key with you?
- 5. sshd_config
-
Using options such as "AllowGroups" to limit the users which are considered valid for logins, regardless of the shell which may be set in /etc/passwd.
Perhaps run the daemon on a non-standard port so that scans only looking at :22 don't detect you're even running SSH.
Most of the things suggested above are covered in manual pages and other articles on this site, I am planning to write up some more in-depth coverage as part of this series, including coverage of sshd_config "Best Practice", cracklib and PAM, authentication using public key, etc.
This is the first article I've written in a long time, hope you enjoyed it.
For reference you might want to take a look at our earlier collection of advice upon Keeping SSH access secure.
A large number of British ISPs also adversely QoS access to anything other than 'standard' ports during peak hours, which generates lots of lag. Boooo.
[ Parent | Reply to this comment ]
I generally find that 22 is not considered a standard port, but for that I have a system setup to do ssh tunnels over port 80. The QoS issue is interesting, but I've never come across it in the US. And I'd probably be fine with lag in return for the freeing the resources consumed by trying to deal with these login attempts.
My advice would be to try changing the port first, and only if you find that to be unacceptable is it worth investigating other options.
[ Parent | Reply to this comment ]
i've never had ssh on a standard port since i got broadband years ago. i've only had one connection ever and that's because i was abusing "http/1.1 connect" to connect to my ssh server through a web proxy. (within an hour an ip address associated with blue coat created a tcp connection on that port, but didn't attempt login because it was probably just seeing what kind of server i had used "http/1.1 connect" for since it wasn't port 443.)
yes, it doesn't block against a determined adversary who will scan all 65535 ports and try to brute force ssh once he finds it, but because there's no noise in the logs, from thousands of bots trying to brute force (or just connect to) my ssh server, you can easily see the real threats.
assuming your adversary gets that far, you can implemented ssh keys (which i only use from my own hardware so as to not allow keylogging or bruteforcing of my key's passphrase) and otp with opie (on "untrusted" hardware, though session hijacking by way of the network socket is possible by a rootkit), eliminating password brute forcing. you can even eliminate brute-forcing root's otp with pam's securetty (brute forcing root's 2048 rsa/dsa ssh key is still a possibility, but one that most people are willing to risk ;-). this requires a two-step process for a valid user to log in as root (ssh to normal user using ssh key or otp, then su to root using password or otp), but requires the attacker to know a user name before even attempting to brute force ssh key or otp.
about ssh not being on port 22 and therefor blocked: most company or public access firewalls i've encountered block everything but http/https, usually through a (transparent) proxy, with the occasional allowance for telnet & ftp, so ssh gets blocked regardless of what port it is on. that's where ajaxterm comes in. ;-)
moving ssh to an off-port is equivalent to moving the safe from outside at the mail box to inside behind the picture on the wall.
[ Parent | Reply to this comment ]
The firewall is able to do some funky stuff also — I use a setup somewhat like this:
iptables -P INPUT DROP
iptables -N ssh
iptables -A ssh -m state --state ESTABLISHED -j ACCEPT
iptables -A ssh -m recent --update --seconds 300 --hitcount 3 -j REJECT
iptables -A ssh -m recent --set -j ACCEPT
iptables -A INPUT -i lo -j ACCEPT
iptables -A INPUT -i tun0 -j ACCEPT
iptables -A INPUT -s $INTERNALNET -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ssh
The ssh-table will now reject the fourth and later attempts to create an SSH connection within five minutes.
Since it can be hard to guess a matching username and password in three guesses, this is good enough for me (root login is not permitted in my setup).
[ Parent | Reply to this comment ]
> The ssh-table will now reject the fourth and
> later attempts to create an SSH connection
> within five minutes.
I do this with my machine too but it does have a downside. It doesn't differentiate between valid and invalid connections. If you do genuinely ssh several times in quick succession to the machine it will still ban you from connecting for a while. This might be because you are opening several terminal windows which run ssh on your desktop (either "ssh host xterm" or "xterm -e "ssh host") or if a script runs ssh to the same host several times.
[ Parent | Reply to this comment ]
Hehe ... I know. :-)
To get around that problem I use VPN. Incoming connections from tun0 and the local network is then accepted unconditionally.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
This downside doesn't exist IIRC when reusing an existing socket...
[ Parent | Reply to this comment ]
iptables -A INPUT -p tcp --dport 22 -i eth2 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -i eth2 -m state --state NEW -m recent --update --seconds 600 --hitcount 5 -j DROP
iptables -A INPUT -p tcp --dport 22 -i eth2 -j ACCEPT
[ Parent | Reply to this comment ]
e.g. AllowUsers bob@*.dsl.pipex.net bob@10.98.23.*
Be careful though, because If AllowUsers is specified, login is allowed only for user names that match one of the patterns. Thus, you have to list all usernames that should be able to ssh to the box, not just the ones you want to restrict to particular hosts.
I always use this, in combination with some other method picked from the ones you mention above (yes, I've tryed them all and use them all in different boxes :))
Thanks for the nice article!
[ Parent | Reply to this comment ]
If you want be cutting-edge, lastest debian package (that actually is not yet in sid) is avaible at the maintainer homepage (myself):
http://bertorello.ns0.it/debian/binary/denyhosts/denyhosts_2.6-3_ all.deb
bye,
Marco Bertorello
[ Parent | Reply to this comment ]
The solution I've settled on is a combination of fail2ban and a restrictive AllowUsers entry in my sshd_config file. I allow only a couple obscurely named accounts to login via SSH (don't totally discount security through obscurity!). I also use fail2ban to block repeated failed login attempts by modifying the server's local firewall. The firewall drops all packets from the attacker, while hosts.deny only affects a few services. Offending IP addresses are only banned for 15 minutes which is long enough to make the script kiddies go away, but prevents all but the most targeted DoS attacks. The 15 minute lockout also prevents me from DoSing myself with my poor typing skills!
While I use PuTTY's Pageant for public key auth all the time at work, I don't like it for accessing my home server. I feel better typing my password every time.
[ Parent | Reply to this comment ]
If your box is compromised, all they have to do, is go through your history to get a list of additional boxes to get access too.
One more for configuring the sshd_config:
Turn off root's ability to log in. While it doesn't stop them from trying, it is one more layer of defense. Because you know no matter what, they shouldn't be able to get in on that vector.
One of the Pros for Denyhosts:
You can get everyone else's block list too. Works great, I've seen failed logins drop to near 0 by syncing with other Denyhosts users. Lots of rejects, but almost no failed logins.
crattis
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
If your box is compromised, all they have to do, is go through your history to get a list of additional boxes to get access too.You mean the box holding the private key, not the server running sshd, right? Of course, the private key itself would be stored on disk with a strong passphrase as well, no? So yes, it could be a cascaded compromise if your local machine is compromised with the key, but it would require the compromising attacker to sort out your key's passphrase as well.
Another option is to offload the key itself to a removable RSA engine. I do this, and detailed how it works with the Cryptoflex eGate (back under debian sarge -- that page is seriously out of date, sorry!). In that case, even if your local machine is compromised, the private key is still inaccessible.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
Sounds like desktop is running Windows, with putty as an ssh client, you have a single key stored on desktop, and you're using pageant for key management. Is this right?
If you have to give pageant a passphrase at startup, that's because it is automatically trying to load your private key, and the key itself is protected with a passphrase. If you tell pageant to unload the key, it should erase all knowledge of the passphrase from its memory.
You also said:
I don't ever remember that box asking for a passphrase to use my keys.Do you mean tools? Do you have a private key stored on tools? If you're using key-based authentication to ssh from tools to other places, you could do it without any private key on tools by simply forwarding your pageant-based authentication. In this case, if your account on tools is compromised, the attacker could read your history and attempt to connect to other hosts as you, but it would only be able to do that while your pageant authentication is forwarded.
I don't know if pageant is capable of prompting you directly each time a forwarded authentication request comes in, but if it can, i recommend that you try that setting. Then even if a compromised remote account attempts to use your forwarded authentication, you'll be aware of it. openssh's ssh-agent is capable of requiring end user confirmation for keys added to it with ssh-add -c, which is pretty handy.
[ Parent | Reply to this comment ]
Desktop (at work) running windows. I use putty (with pageant) to go to tools, where I maintain around 50 or so linux servers. tools does have it's own private key. I occasionally access tools from my home. When I do that, I don't use a key, but the password. home is a linux box. So forwarding from pageant won't always work.
I'll look into the the ssh-agent. Thanks. I also didn't mean to make it sound like a compromised box was a regular occurrence, just a chance thing that could happen.
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
Assuming tools is running OpenSSH and your private key is stored in the usual location, You can set up a passphrase on your private key with:
ssh-keygen -p ~/.ssh/id_rsa
Even better would be to remove your private key from tools entirely, and just use forwarded authentication (or the ProxyCommand option to OpenSSH, which deserves a whole other post). With the key removed from tools, the only way that a compromise of tools could cascade is if the attacker piggybacks on your existing forwarded authentication session.
Since you connect to tools from at least two machines (desktop and home), you'd need to either put a copy of your private key on each machine (downside: each copy is yet another copy that could be compromised) or put the key on a removable device that you could take with you and use on any machine (downside: if you lose that device, you could be pretty screwed!).
You might also want to consider the idea of a Trusted Physical Console -- if you always work physically from the same machine, you can just keep your keys on that machine (and take it with you), and forward the agent authentication or ProxyCommand hop when needed.
[ Parent | Reply to this comment ]
I had problems putting key on mine from the outside. But now it is all OK!
GPG can work with the help of: http://gnupg-pkcs11.sourceforge.net/
--
cstamas
[ Parent | Reply to this comment ]
CONS : Hard to master, like public keys
PROS : Secure and you can easily compute your one time password with a web page (javascript).
Just remember to add [:skey] to your login and compute them ;)
[ Parent | Reply to this comment ]
I permit either PKI or OPIE/SKEY authentication from remote. I do not permit password authentication at all. With this configuration it is incredibly unlikely that a brute-force attack would ever work. OPIE/SKEY clients exist for many platforms, including my Nokia mobile phone.
If on the console, it will still prompt for OPIE/SKEY but also accept a password as a failsafe.
I also rate-limit SSH connections per second on the firewall, and use fail2ban.
I found that I get some brute-force attacks on postfix as well. This is more difficult to secure and I rely on fail2ban in this case. If the level of brute-force attacks increases, I'll have to think of something else.
[ Parent | Reply to this comment ]
How do you manage to have a different authentication scheme from console than via openssh?
Can you post your sshd_config file (or other related configs)?
[ Parent | Reply to this comment ]
Authentication methods for various things can be controlled with PAM. I'm using Debian, so this might need significant modification for your system. My /etc/pam.d/ssh looks like (the part that's different than default):
auth sufficient pam_opie.so
auth required pam_deny.so
You'll want to comment out "@include common-auth" once you know that OPIE is working.
I've also modified /etc/pam.d/common-auth to permit both OPIE (first) and then password (second):
auth sufficient pam_opie.so
auth sufficient pam_unix.so nullok_secure
auth required pam_deny.so
In /etc/ssh/sshd:
PubkeyAuthentication yes
ChallengeResponseAuthentication yes
PasswordAuthentication no
UsePAM yes
I realize this is brief but hope it helps, or at least gives you an idea of what to look for.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
[ Send Message | View dkg's Scratchpad | View Weblogs ]
Compounding the problems, port knocking is fragile in the presence of firewalls between you and the remote host. If any firewall or filter decides to block any of the ports you've tried to use, your port knocking will fail, cutting you off from the remote machine.
If your SSH daemon's authentication is weak enough that the gain provided by port knocking seems significant, you should probably instead spend more time thinking about how to improve the choices you've made for SSH authentication (reading this article and the comments here is a good start!).
[ Parent | Reply to this comment ]
Port Knocking adds very little true security to an SSH service.
We should think about what *kind* of security we need -- that is, what sort of threat are we defending against?
Against an attacker who has identified my computer and is determined to compromise my specific computer, your comments are correct. But I would say that that scenario is rare. Much more common, as we see from the article and the comments, is automated scanning. A bot is not going to try to figure out if a target is using port knocking, or analyze traffic to that target to find a port knock sequence -- it's going to hit port 22, and move on when it finds the port is closed.
[ Parent | Reply to this comment ]
That'll protect you just fine until a smarter variant comes along, or until you annoy a 'real' human who actually knows what they're doing.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
# Change to no to disable tunnelled clear text passwords #PasswordAuthentication yes PasswordAuthentication nofollowed by `/etc/init.d/ssh reload`. This means that you can access this Box ONLY via an entry in ~/.ssh/authorized_keys.
I do this on ALL Boxen which have an open ssh port on an untrusted network. (and I know very few trusted nets ;-)
Also, while we're at it, if you are behind a firewall which you control, (IPCop at home) then you might want to consider mapping the "visible ports" to something other than 22. You can do this for several Boxen if you wish; I have used the convention "22+last-byte-of-IP" at times. On the road I could call home with `ssh -p22123 user@outsideaddr.ISP.TLD` and land on port 22 of the Box x.y.z.123
keep up the good work
//rhi
[ Parent | Reply to this comment ]
I have written a pam module to block repeated failed attempts from the same IP address. (it is used in production on couple of servers for about a year).
You can find it if you search for "hostblock" on http://rubyforge.org/ .
It is PITA to install and needs more testing.
Vlad
[ Parent | Reply to this comment ]
Setting this option to "no" will disable the access to user root. This is very valuable in my (humble) opinion, because this is the only user whose name is always known to exist on your system by the attacker.
So, by combining "PermitRootLogin" and "AllowGroup" it is very easy to restrict access to SSH to one or several real users (meaning not system with standard and predictable logins as 'www-data', 'ftp', etc...), whose logins are very unlikely to be found by an attacker.
So the brute force attack has to found the password AND the user, which significantly reduces chances of success because of course OpenSSH gives no details when refusing authentication: the attacker cannot know if it is the password or the login which is wrong.
[ Parent | Reply to this comment ]
May I ask what are the advantages of such a scheme, if the user chooses fairly secure passwords and doesn't use them for unsecure protocols?
I can see many disadvantages: having to carry around a USB key with your private keys, having to set up authentication forwarding or such (if you want to use ssh inside another ssh session), but no actual benefit.
[ Parent | Reply to this comment ]
cd ~/.ssh
touch rc
edit rc with an editor:
-----------------------
machine=`cat /etc/hostname`
user=`whoami`
address=$SSH_CONNECTION
ip=`echo $address | awk '{print $1}'
date=`date +'%d/%m/%Y %H:%M'`
echo -e "machine\t: $machine\n\
user\t: $user\n\
address\t: $address\n\
date\t: $date"\
| mail -s "ssh: $user@$machine $ip $date" mail@domain.com
acemi
[ Parent | Reply to this comment ]
ip=`echo $address | awk '{print $1}' above
misses the closing `
Gijs
[ Parent | Reply to this comment ]
* Higher load. Brute force costs quite some cpu time which can be a DoS for embedded systems or at least 10% cpu usage for normal old systems.
[ Parent | Reply to this comment ]
[ Parent | Reply to this comment ]
I am a KISS person by nature so I decided to go my own way. I set up a list of things I would try but I would try one thing at a time to see what effect it would have. The first thing on my list was to simply change the port number (you mentioned this at the end of your article).
FYI, that is as far as I got on my list. Attacks on my machine through SSH have virtually disappeared altogether.
It is interesting to note that none of the articles I read during my research mentioned this step yet it was extremely effective and the simplest solution by far. The reason why it is effective is probably because the scans that are being done are automated, most probably by people who don't really understand what they are doing, or automated scans from machines that have been pwned. Nobody is portscanning me and trying to figure out what I am running behind each port. As a homeuser I am simply not worth the effort.
So my opinion is to try this first and foremost and then monitor your logs. I am certain that you will see these attacks die down. However, if you are looking into these things for something other than home network then make sure you dig deeper into the security solutions out there.
/David
[ Parent | Reply to this comment ]