Rescuing a system with massively broken filesystem permissions

Posted by Steve on Thu 21 Dec 2006 at 09:30

There are times when people make mistakes, and manage to do crazy things to working systems. It wasn't so long ago that a hasty deletion caused me all kinds of problems. Recently I read of an unfortunate sysadmin who managed to recursively change permissions on their root filesystem - and here is my attempted solution.

The problematic command was:

(none):~# chmod -R 777 /

Here the entire filesystem, and all mounted partitions, were modified to be world-writable. One immediate consequence of this is that ssh logins fail:

skx@lappy:~$ ssh -l root
ssh_exchange_identification: Connection closed by remote host

Looking in /var/log/auth.log upon the host we can see the cause of this failure:

error: Could not load host key: /etc/ssh/ssh_host_rsa_key
error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
error: @         WARNING: UNPROTECTED PRIVATE KEY FILE!          @
error: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
error: Permissions 0777 for '/etc/ssh/ssh_host_dsa_key' are too open.
error: It is recommended that your private key files are NOT accessible by others.
error: This private key will be ignored.
error: bad permissions: ignore key: /etc/ssh/ssh_host_dsa_key
error: Could not load host key: /etc/ssh/ssh_host_dsa_key
fatal: /var/run/sshd must be owned by root and not group or world-writable.

These errors are fixable fairly easily since we still have an open root session:

(none):~# chmod 600 /var/run/sshd
(none):~# chmod 600 /etc/ssh/ssh_host_*

Since we have root access we can now fix things up, and the availability of remote ssh access is a nice bonus. However these remote logins cannot become root themselves just yet - because su and sudo have lost their setuid flags:

skx@(none):/$ su
setgid: Operation not permitted

We can fix this fairly easily, again using our existing root login:

(none):~# chmod 4755 /bin/su /usr/bin/sudo

Now we can login remotely and become root. So the system is 90% fixed!

But what if we didn't have root access? What if we could only login via a normal user account, via the console perhaps?

Lets try that now.

(none):~# chmod -R 777 / 
(none):~# exit

Remote logins via ssh will have become broken again - how can we fix things now? Suddenly we can't fixup permissions as we're not root, and we can't become root since the binaries are no longer setuid.

It seems that our first job is to gain root access. Whilst there are many things we could try (such as emptying the password fields in /etc/passwd - possible now all files are world-writable) first solution to occur to me was to abuse a daemon which is already running and which has permissions to do root-type jobs. In a word: cron.

Since the system has various php4 packages installed we can modify the cronjob which has been installed:

echo "*/1 * * * * root cp /bin/sh /tmp/sh && chmod 4755 /tmp/sh" > /etc/cron.d/php4

Waiting a minute finds us a setuid shell in /tmp and we can use that to become root and fix things as we did previously:

skx@uml:~$ /tmp/sh                                  
sh-3.1# id
uid=1000(skx) gid=1000(skx) euid=0(root) groups=1000(skx)

Having writable files will allow a lot of different attacks against a system, cron is just the most obvious. The important thing is to not panic, and do anything hasty such as rebooting or exiting your root shell!



Posted by spiney (85.127.xx.xx) on Thu 21 Dec 2006 at 14:09
Damn, the moment I thought the interesting parts come, the article was over. Not that I was ever in the situation (and I hope I never will), but what would be the quickest way to get the permissions set up properly again? Reinstalling packages fixes a lot, but I'd be interested to know how people would handle this.

Kind regards
Wolfgang Karall
Debian GNU/Linux on an IBM Thinkpad T43p

[ Parent | Reply to this comment ]

Posted by Steve (80.68.xx.xx) on Thu 21 Dec 2006 at 15:02
[ View Steve's Scratchpad | View Weblogs ]

I guess it did just tail off a little, but the hard parts were done. Once you can login as root remotely then you're easily able to do the site-specific fixups from the comfort of your desk.

If it were me I'd just restore from backups which would incidentally fix the permissions!

Still if you have another working system I guess the obvious thing to do is to write a simple script to output a file containing the path + permissions of every file on the system. Then another to read in that file and apply the fixups.

I remember doing that once in the past. Something like this simple script would do the job:

#!/usr/bin/perl -w

use strict;
use File::Find;

find(\&wanted, "/");

sub wanted
    my $file = $File::Find::name;
    my $mode = (stat($file))[2];
    printf "%04o, %s\n", $mode & 077777, $file;

This will give output such as this:

40755, /
0600, /.bash_history
0600, /.viminfo
0600, /.lesshst
0600, /initrd.img.old
0644, /vmlinuz.old
40755, /lost+found
40755, /home

The script to restore those permissions is left as an excercise ;)


[ Parent | Reply to this comment ]

Posted by mvanbaak (80.126.xx.xx) on Thu 21 Dec 2006 at 17:57
This is part of my daily backup scritpt:
getfacl -R / > /home/backups/acl-`date +%d-%m-%Y`.txt

with setfacl you can recover this :)

[ Parent | Reply to this comment ]

Posted by spiney (85.127.xx.xx) on Fri 22 Dec 2006 at 07:49
I completely missed that setfacl/getfacl are useful on non-ACL-enabled systems, but this hint is very helpful. Backup scripts adapted, thanks a lot!

Kind regards
Wolfgang Karall
Debian GNU/Linux on an IBM Thinkpad T43p

[ Parent | Reply to this comment ]

Posted by xrat (128.130.xx.xx) on Wed 27 Dec 2006 at 20:11
On my box (ext3 on SW-RAID1) running acl 2.2.23, getfacl bailed out with "getfacl: /: Not a directory" after processing /proc. getfacl from acl version 2.2.41 (testing) works fine. -- Andreas

[ Parent | Reply to this comment ]

Posted by yarikoptic (165.230.xx.xx) on Thu 21 Dec 2006 at 16:12
> ldd /bin/sh =>  (0xffffe000) => /lib/ (0xb7eaf000) => /lib/tls/i686/cmov/ (0xb7eaa000) => /lib/tls/i686/cmov/ (0xb7d76000)
        /lib/ (0xb7f1e000)

So if you screw up your dyn libraries somehow (move them aside while trying to localize possible intrusion detected by the simple md5 sum check... don't ask) -- then you are doomed to fail starting up any new application which relies on dyn libraries, including /bin/sh.

Whenever I got into such situation, I did very good thing -- I didn't panic, took beer and a cigarette and just relaxed to become sober in my actions ;-) (to don't say that I was ~6kmiles away from the system, so I just had ssh/VNC access... where obviousely no new connections could be setup)

Lyrics aside, keep some staticly compiled shell readily available:

sash - Stand-alone shell
zsh-static - A shell with lots of features (static link)
bash-static - The GNU Bourne Again SHell (static version)

sash installation (may be others as well) optionally does a handy thing - creates sashroot user which uses sash as his shell ;-) so if system is %%ed you still have a chance to login remotely (Surely you better you need to restrict access to that account...)

[ Parent | Reply to this comment ]

Posted by Anonymous (84.45.xx.xx) on Thu 21 Dec 2006 at 18:02
"It seems that our first job is to gain root access."

Boot from rescue CD, pick unused console, "chroot /mnt/rootfs ; passwd".

Root doesn't have to give the old password to set a new one.

I'm not sure one should rely on having insecure temporary files around the place to become root.

I'm not sure leaving a system with warped file permissions up is a good idea either, but hey Linux and Unix are remarkably robust to such cock-ups.

Perhaps the next article should be on the U-Kuang program, and how to spot when you've left stupid file permissions laying around.

[ Parent | Reply to this comment ]

Posted by elivs (202.78.xx.xx) on Fri 22 Dec 2006 at 17:54
Great article. We all break our systems from time to time !

What I'd like to know is once you have root access is there a simple way to get apt/aptitude to refresh all the permissions in /usr /sbin etc?

The same question applies to the situation where you accedently delete part of /usr. Is there a way to get apt to verify every install package?

<disclaimer> I probably should just use google to answer this question, but if you could post and answer here it would be a good resource for people.<disclaimer>


[ Parent | Reply to this comment ]

Posted by Steve (62.30.xx.xx) on Fri 22 Dec 2006 at 18:03
[ View Steve's Scratchpad | View Weblogs ]

Not directly; although you could run "apt-get install --reinstall [package1] [package2] .." to do it.

Failing that you could install the acl package and use getfacl + setfacl to create/restore from a backup as suggested by mvanbaak already.

In the case of missing files then the "--reinstall" trick might work, but the obvious answer is "restore from backups". If you don't currently have backups you'll have learnt how important they are for the future ;)


[ Parent | Reply to this comment ]

Posted by Anonymous (199.82.xx.xx) on Tue 9 Jan 2007 at 15:02
I did something far far worse then this once, and it was a life lesson on why to use relative paths. I typed "rm -rf /" and then sneezed. I hit control C with a quickness. However, you know what the first things to go where? /usr/bin, /bin, and /sbin. Needless to say I was humped and had to simply rebuild by my box. Luckily the delete did not make it to the partition that kept all of my personal stuff on. My friends still give me grief to this day about that one.

[ Parent | Reply to this comment ]

Posted by tarunbhatia (64.185.xx.xx) on Wed 24 Oct 2007 at 23:14
Thank God for this post otherwise I would have to reinstall the entire OS on my linux box again. However in my case, I had already rebooted my machine and was no longer able to login as root. Fortunately, I had knoppix on a CD. When I booted in to that for some reason knoppix mounted my Hard Drive as a read only. To change the permissions on that, all I had to do was unmount then remount the drive. After that, I just followed the instructions and it has been working perfectly fine so far ever since.

[ Parent | Reply to this comment ]

Posted by Anonymous (71.146.xx.xx) on Thu 21 Feb 2008 at 19:51
I was taking care of a sizable number of Unix systems at work, a few years back, and one of my desktop users who used to do things in strange ways.One day he totally messed up his SunOS 5.x system by a bad chmod -fR
command on the root filesystem.
I wanted to just put the modes back, by duplicating them from a similar
desktop system ... just the items that matched name and location.
I couldn't figure out how to do that using rsync at the time, not sure if
it was feasible.So I wrote a shell script that accomplished it, I don't remember if it was based on cpio or something else. Now I'm thinking that I may have written some perl code that traversed the system, operating on each directory that existed on both sides.Anyhow, I think that it's possible to use rsync to derive just the modes, if you have a second system that you want to copy modes from (assuming that you've fixed the broken system enough to make the basic commands and networking run, and you just want to clean up the rest of the system areas). Larry

[ Parent | Reply to this comment ]

Posted by Anonymous (203.89.xx.xx) on Fri 7 May 2010 at 04:43
Reinstalling: As root
apt-get --reinstall install $(dpkg -S /usr/bin/* | awk ' { print $1 } ' | sort -u | sed -e 's/:$//')

Should do the package reinstall.
I had a remote login with ssh-keygen ability from my master server. Otherwise there would have been no ability to change to root user.

[ Parent | Reply to this comment ]

Posted by Anonymous (24.132.xx.xx) on Sat 15 Oct 2011 at 22:10
When searching for a solution I also found another trick to re-gain root access at least on Ubuntu.

The command issued was to allow group read / write permissions. So this did really destroy my system, but not so much that I could easily regain root access via the methods mentioned in the post (I had no root shell).

Someone on another forum mentioned pkexec. Apparently pkexec is not that choosy about file permissions and will let me get a root shell as long as I am logged in as a user in the admin group. This allowed me to get root access with the running ssh session.

Thanks for the original post, it helped!

[ Parent | Reply to this comment ]

Posted by Anonymous (92.249.xx.xx) on Sat 23 Jun 2012 at 14:54
find / -type d -print0 2>/dev/null |xargs -0 ls -ald|awk -F " " {'print $9" "$3" "$4" "$1'} >all_perms_dirs
find / -type f -print0 2>/dev/null |xargs -0 ls -al |awk -F " " {'print $9" "$3" "$4" "$1'} >all_perms_files

+ perl script(which make bash script for recovery permissions).

[ Parent | Reply to this comment ]

Sign In







Current Poll

What do you use for configuration management?

( 117 votes ~ 0 comments )