New User? Register here - Existing Users: Username: Password: [Advanced Login]

 

 

Current Poll

What language for system administration do you use ?







( 731 votes ~ 3 comments )

 

Weblog entry #19 for lykwydchykyn

See all weblog entries made by lykwydchykyn.

XML logo

#19
Why do upgraded servers not come back from reboot?
Posted by lykwydchykyn on Mon 14 Feb 2011 at 17:13
Tags: none.
I've been upgrading my debian servers to squeeze over the last few weeks. Everything goes fine when I upgrade servers on the VM system, or here in the building.

For some reason, every time I do a remote upgrade, the system fails to come back up after reboot. Every time. I have to go out or send someone out to power off the server and cold boot it. After that it works fine.

I can't seem to reproduce the problem back at the office, and I'm getting nervous because the two I have left to upgrade do DHCP and DNS for some fairly large (and distant) facilities.

Can anyone hazard a guess as to why this is happening (apart from murphy's law)?
[ 4 Comments | Add Comment | XML logo ]
<<< Previous

 

Comments on this Entry

#1
Re: Why do upgraded servers not come back from reboot?
Posted by Steve (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Tue 15 Feb 2011 at 09:05
[ Send Message | View Steve's Scratchpad | View Weblogs ]

If the machines haven't been restarted "recently" they might be coming back - but very very slowly as they run an fsck on boot?

Using a serial console will be the way to see what is happening; you might consider trying the network-console thing the kernel has to see remote messages via udp?

Steve

[ Parent | Reply to this comment ]

#2
Re: Why do upgraded servers not come back from reboot?
Posted by lykwydchykyn (72.237.xx.xx) on Tue 15 Feb 2011 at 14:51
[ Send Message | View Weblogs ]
I've thought about that, but sometimes I've waited several hours (overnight) and they don't come back up. When we get there, they just have a black screen.

I'm intrigued about the serial console/UDP -- never heard of it. Do you have a link?

[ Parent | Reply to this comment ]

#3
Re: Why do upgraded servers not come back from reboot?
Posted by Steve (2001:0xx:0xx:0xxx:0xxx:0xxx:xx) on Tue 15 Feb 2011 at 14:55
[ Send Message | View Steve's Scratchpad | View Weblogs ]

You'd be looking something like this.

Though in your case I guess you'll be needing to add the module directly to the kernel command line, via grub/lilo, to make sure it loads immediately. So this guide might be more appropriate to you.

Steve

[ Parent | Reply to this comment ]

#4
Re: Why do upgraded servers not come back from reboot?
Posted by simonw (78.33.xx.xx) on Tue 15 Feb 2011 at 15:28
[ Send Message | View Weblogs ]
I was going to add a comment about sysrqd as a possible help.

I had servers not unmounting nicely at end of upgrade, I assume the change to the boot procedure or some such leaving stuff running that wouldn't normally. In my case most are either local or virtual, so I have a remote console.

[ Parent | Reply to this comment ]

 

 

Flattr