/boot is 100% full

[samengr]

Hi Linux gurrus

hope you all will be doing well.

Well I am facing a problem first time in my sys admin experience. I am using RHEL-4, I am unable to access /boot partition. when i try to access this partion or want to show its contents, It seems /boot is corrupted. see the following outputs

#ls /boot/
9?^?????tdÜ°??????4?çj?

#df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
145G 98G 40G 72% /
/dev/sda1 1.4T 1.4T 0 100% /boot
none 501M 0 501M 0% /dev/shm

Note: its showing the wrong entries /dev/sda1 (1.4T) and its 100% full.

#fdisk -l

Disk /dev/sda: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 9726 78019672+ 8e Linux LVM

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 9726 78124063+ 8e Linux LVM

Can any body help me whats wrong with /boot? Number of applications are being run (24x7) on this server. I cant reboot it and cant interrupt the running applications. Can any body tell me what should i do now?

Server's info:

#cat /etc/redhat-release
Red Hat Enterprise Linux ES release 4 (Nahant Update 5)

# uname -r
2.6.9-11.ELsmp

waiting for an early response.

Thanks
Sohail

[valan]

Have you fscked it? (I'm not trying to make a joke here) Perhaps, if unable to fix the fs, you might have to recreate the partition/fs, install the kernel and boot loader on it manually to fix it. If you don't have another copy of your current kernel or config, you will have to build another one or temporarily use the default until another can be built. Luckily it's only /boot and not something non replacable.

[samengr]

Thanks for the reply.
No, I didnt check the fsck. I just wanted to discuss before doing any thing because this is the most important server cant afford to loose the running applications.
Can you please help me out? do tell me step wise what should i do now? from where to troubleshoot?
1- fsck /dev/sda1 ??
2- can i recover /boot folder with all its contents? As i have backup of grub.conf and other imp boot files in the /boot folder no where to go?
I have other RHEL4 servers but as i joined new here, dont know are they have same boot setup or not?
thanks for you time and waiting for your help.

Sam

[sdimhoff]

1.) I think that looking at what fsck throws back at you is the first step.

The "good" thing about your problem is that there shouldn't be anything in the boot partition that is irreplaceable or should be necessary for any of the currently running applications (I assume that most of the time it isn't even mounted.). Additional note, if you just want to see what fsck would output you can give it the N flag. fsck -n (unless it is reiserfs or minx)

2.) If fsck fixes the problem then great, you are done (although you should check files to be sure that fixing the drive errors didn't screw up any data). If not, then I think Valan was on the right track. You may be forced to reformat the /boot partition then reconfigure the bootloader and copy over your kernel.

If worst comes to worst, the next time the server restarts, it won't boot. In that case you can just use a liveCD to fix it.

[samengr]

# fsck -n /boot
fsck 1.35 (28-Feb-2004)
fsck.ext3: Unable to resolve 'LABEL=/boot'

Now?

[samengr]

can any of you tell me step-by-step what can i know? or can recommend any tutorial/pdf for this issue? I know if i rebooted the server it wouldnt boot it up

[MikeTbob]

I think you need to use the device point and not the mount point.
fsck -n /dev/sda1
EDIT: The best info on fsck is the man pages, open a terminal window and type this command
man fsck
Try Google also.
fsck.ext3 - Google Search

[samengr]

# fsck -n /dev/sda1
fsck 1.35 (28-Feb-2004)
e2fsck 1.35 (28-Feb-2004)
Warning! /dev/sda1 is mounted.
Couldn't find ext2 superblock, trying backup blocks...
Superblock has a bad ext3 journal (inode .
Clear? no

fsck.ext2: Illegal inode number while checking ext3 journal for /boot


Now what next ??

[MikeTbob]

I would try unmounting the drive, then try fsck again, if you get the same errors, then maybe it's time to think that it really is trashed and needs to be rebuilt. I am no expert, so you need to use your best judgement here.

[sdimhoff]

Sorry I wasn't specific on the commands you needed, but it looks like Mike clarified things.

Additional Advice:

From what you've said, I think you were right in your estimation that the server will NOT reboot if you tried now. The system will run fine until the next reboot (unless you have a failing hard drive) therefore you should plan out your attack and notify your users BEFORE the downtime will begin.

It also seems like you don't really know what sort of plan of attack you should do so I've laid out what I would do. I want to be clear that the following is simply what I would do, and that you have to be ready to deal with the consequences if this doesn't work out. (Of course, you can always post new problems here along the way!)

1.) Do the fsck check but let it fix the errors.
umount /boot
fsck /dev/sda1

2.) IF the fsck continues to fail then you have to be sure that no users are trying to access your files, because this will require a reboot AFTER re-making the boot partition.

3.) Reformat /dev/sda1 (this will wipe the boot partition, so you should be sure it is what you want to do.)

4.) Mount the new boot partition and reconfigure the bootloader and copy over your kernel. (I'm sure there is documentation for RHEL server)

5.) Be sure that your bootloader is pointed to your newly copied over kernel, and only after you are either sure that things are correct, or your users know there will be downtime can you reboot.

6.) If you have configured your bootloader incorrectly and have rebooted, you can always boot to a liveCD to fix it.

7.) If problems continue to occur then this may be a symptom of a harddrive failure.

Good luck and remember that you have plenty of time to fix this (the system is currently running and shouldn't need the boot partition until reboot). It is always difficult to fix problems when a server is needed by many people, so you have to plan and notify your users if there will be downtime.

__

Also, if any other people spy an error in what I've written then please feel free to correct me.