How many hours did you spend updating systems made vulnerable by the Debian OpenSSL bug(DSA-1571)?
Submitted by emeitner on Tue 3 Jun 2008
| None |
   17% | 389 votes |
| less than 1 hour |
36% | 815 votes |
| 1-5 hours |
25% | 555 votes |
| 6-10 hours |
8% | 182 votes |
| 11-20 hours |
3% | 82 votes |
| 21-30 hours |
1% | 38 votes |
| 31-40 hours |
0% | 17 votes |
| more than 40 hours |
6% | 138 votes |
| Total 2218 votes |
Fortunately i only had 2 severs and 5 workstations suffering from that vulnerability.
[ Parent ]
Posted by Steve (82.41.xx.xx) on Wed 4 Jun 2008 at 11:47
[ Send Message | View Steve's Scratchpad | View Weblogs ]
[ Send Message | View Steve's Scratchpad | View Weblogs ]
I can only concur.
But despite having CFEngine setup on 100-150 machines there were still many hours spent testing things, and the ongoing time spent validating and accepting new SSH keys when re-connecting to updated machines.
[ Parent ]
Posted by Anonymous (84.105.xx.xx) on Tue 3 Jun 2008 at 20:43
This really was a bad one. Updating the systems wasn't our biggest problem. That took like 10 minutes for all of our servers. But finding and replacing all of our ssh-keys proved to be a bigger problem.
This all had one positive side tho... While replacing all the keys and testing the scripts again I also found some bugs which I then fixed ;-)
Roedie
This all had one positive side tho... While replacing all the keys and testing the scripts again I also found some bugs which I then fixed ;-)
Roedie
[ Parent ]
An excellent idea for a poll. I look forward to all the extrapolations... how many Debian systems are there in total?
My main worry was to need to promptly advise users, and also former users who may still have vulnerable keys forgotten about in authorized_keys files on their servers, that they needed to take action.
The next most painful bit was dealing with SSL certificates. This is one of those subjects that I have to re-learn every time I deal with it, and it probably took me half a day or so to be certain that I had made the right changes. You might also like to ask people how much they had to pay for new certificates.
Worryingly, as far as I can see only about one user in three has actually acted on my email telling them that their keys were vulnerable. If this is generally true, then there are a hell of a lot of vulnerable systems still out there. I have not yet seem any attacks attempting to exploit this - has anyone? - which surprises me, since I see a lot of ssh password attacks. If such attacks do start, I think many machines will be compromised.
I do also worry that all those people who've answered "less than 1 hour" here may have not thoroughly understood the implications of this situation. Do those people just have a single machine, not internet-facing?
My main worry was to need to promptly advise users, and also former users who may still have vulnerable keys forgotten about in authorized_keys files on their servers, that they needed to take action.
The next most painful bit was dealing with SSL certificates. This is one of those subjects that I have to re-learn every time I deal with it, and it probably took me half a day or so to be certain that I had made the right changes. You might also like to ask people how much they had to pay for new certificates.
Worryingly, as far as I can see only about one user in three has actually acted on my email telling them that their keys were vulnerable. If this is generally true, then there are a hell of a lot of vulnerable systems still out there. I have not yet seem any attacks attempting to exploit this - has anyone? - which surprises me, since I see a lot of ssh password attacks. If such attacks do start, I think many machines will be compromised.
I do also worry that all those people who've answered "less than 1 hour" here may have not thoroughly understood the implications of this situation. Do those people just have a single machine, not internet-facing?
[ Parent ]
Posted by emeitner (216.170.xx.xx) on Wed 4 Jun 2008 at 03:33
[ Send Message | View emeitner's Scratchpad | View Weblogs ]
[ Send Message | View emeitner's Scratchpad | View Weblogs ]
My guess was 10-15 hours. I had 7 servers affected(2 clusters of two nodes, plus three others). Four other Sarge servers had public keys that needed replacing. Lots of SSH keys used for automated systems needed replacing and the systems needed to be tested to ensure nothing was missed. Two SSL certificates for public web servers needed to be reissued. Fortunately I am the only person who actively uses SSH in our organization so there were few interruptions for users.
[ Parent ]
Posted by Anonymous (67.88.xx.xx) on Thu 5 Jun 2008 at 17:45
Sarge was not effected. Any keys you made with Sarge and before should be fine. Not sure why you had to change those? Because we are so slow to upgrade we pretty much got lucky since the majority of what we run is Sarge.
http://www.us.debian.org/security/2008/dsa-1571
"The first vulnerable version, 0.9.8c-1, was uploaded to the unstable distribution on 2006-09-17, and has since that date propagated to the testing and current stable (etch) distributions. The old stable distribution (sarge) is not affected."
http://www.us.debian.org/security/2008/dsa-1571
"The first vulnerable version, 0.9.8c-1, was uploaded to the unstable distribution on 2006-09-17, and has since that date propagated to the testing and current stable (etch) distributions. The old stable distribution (sarge) is not affected."
[ Parent ]
Posted by emeitner (69.129.xx.xx) on Fri 6 Jun 2008 at 04:17
[ Send Message | View emeitner's Scratchpad | View Weblogs ]
[ Send Message | View emeitner's Scratchpad | View Weblogs ]
Sure, but the sarge servers had accounts with compromised public keys in ~/.ssh/authorized_keys. These needded to be replaced.
[ Parent ]
I had to upgrade three desktops, a laptop and two servers. The upgrade was almost automatic and didn't take any effort. My SSH keys were actually generated by PuTTY not OpenSSH so in theory they were okay - turns out a lot of people in my LUG were in the same situation.
I did take the opportunity to replace all my keys with strong 2048-bit keys, so except the machines that were switched off at the time, it was all done within a few hours.
--
"It's Not Magic, It's Work"
Adam
I did take the opportunity to replace all my keys with strong 2048-bit keys, so except the machines that were switched off at the time, it was all done within a few hours.
--
"It's Not Magic, It's Work"
Adam
[ Parent ]
Oh geez, hundreds of servers on dozens of networks... this was a killer. Good thing I could script much of it.
[ Parent ]
Posted by Anonymous (125.236.xx.xx) on Thu 19 Jun 2008 at 11:15
I only spent a couple of minutes on it. I only have one (affected) Debian machine (Ubuntu) which is largely just used as a workstation for myself.
I wasn't thorough and I likely missed one or two of the things (yeah I am very slack). SSH is fine though, and even that can't be reached from the internet. I'll finish the job one of these days.
I wasn't thorough and I likely missed one or two of the things (yeah I am very slack). SSH is fine though, and even that can't be reached from the internet. I'll finish the job one of these days.
[ Parent ]
Posted by Anonymous (80.94.xx.xx) on Wed 25 Jun 2008 at 20:17
Between 6 and 10 hours . Too much servers...
[ Parent ]
Posted by Anonymous (220.233.xx.xx) on Fri 27 Jun 2008 at 08:39
So glad I only had a total of 5 servers and 2 or 3 desktops effected. And of those, 0 critical issues relating to it require testing. :-)
The only problem I hit was I had to regenerate keys for logging into my home server from my home and work desktops... Total time of about 30 minutes because I'd forgotten how to do it.
The only problem I hit was I had to regenerate keys for logging into my home server from my home and work desktops... Total time of about 30 minutes because I'd forgotten how to do it.
[ Parent ]