One day, you’re happily typing away on a production server. You want to clean up /tmp so you can mount it with tmpfs instead of normal on-disk stuffs. You proceed:

root@ubuntu:/# ls /tmp/
dbconfig-generate-include.sed.fc6130  dbconfig-generate-include.sed.hu7261  dbconfig-generate-include.sed.k26205

Ok, on to the action:

root@ubuntu:/# rm *
rm: cannot remove `bin’: Is a directory
rm: cannot remove `boot’: Is a directory
rm: cannot remove `dev’: Is a directory
rm: cannot remove `emul’: Is a directory
rm: cannot remove `etc’: Is a directory
rm: cannot remove `home’: Is a directory
rm: cannot remove `initrd’: Is a directory
rm: cannot remove `lib’: Is a directory
rm: cannot remove `lost+found’: Is a directory
rm: cannot remove `media’: Is a directory
rm: cannot remove `mnt’: Is a directory
rm: cannot remove `opt’: Is a directory
rm: cannot remove `proc’: Is a directory
rm: cannot remove `root’: Is a directory
rm: cannot remove `sbin’: Is a directory
rm: cannot remove `srv’: Is a directory
rm: cannot remove `sys’: Is a directory
rm: cannot remove `tmp’: Is a directory
rm: cannot remove `usr’: Is a directory
rm: cannot remove `var’: Is a directory

You can summarize the next few seconds as “………. OMG OMG OMG OMG OMG”, while turning pale.

root@ubuntu:/# ls
bash: /bin/ls: No such file or directory

It’s a goner. Luckily i didn’t do rm -rf, because that would have FUBAR’ed the server fully. What happened is that /lib64 and /lib32 ( which are symlinks to /lib and /emul/ia32-linux/lib respectively ) got removed, effectively rendering you unable to execute any command ( because everything depends on /lib64/ld-linux-x86-64.so.2 ). But no worries, there is one thing that is statically linked on your system ( meaning, not requiring any libs from /lib64, thus still working ) and that’s the /lib/ld-linux-x86-64.so.2 file. By using this file, you can force your ln command to look into /lib for its libs instead of ( the now missing ) /lib64.

root@ubuntu:~# /lib/ld-linux-x86-64.so.2 –library-path /lib /bin/ln -sf /lib /lib64

Which re-creates the /lib64 symlink, reviving all commands! Then you can create the /lib32 symlink again with the normal ln command. Crisis averted ;)

Hopefully this will help others in the same situation ;)

Remember kids, working as root is evil! ( although it couldn’t help me in this situation since i needed to be root to clean the /tmp. )