rhel6 boot issue after FS went RO

[ltodd2]

Im hoping one of you can help. We have a RHEL6 server installed on our vmware system. The other day the storage to the ESX servers was lost for a short time and the RH server made the filesystem read only. We have rebooted the server but now it wont boot. It shows the red load bar which does make it the the end but slow and thats it. Im out of ideas to try so am hoping someone here may have an idea on how to kick it into life.

Thanks

Lee

[Rubberman]

Try recovering the file system with fsck. I'm not sure how you would do that with an ESX server, unfortunately. Now, the question is, what caused the outage originally, so you can deal with this problem in the future?

[HROAdmin26]

The RHEL system is a virtual machine.

How would you fix this issue if RHEL was installed on bare metal?

1) Boot to single-user mode and run fsck.
2) Boot from other media (CD, USB, etc.) and attempt to mount the volume / use fsck where needed.

[ltodd2]

Rubberman and HROAdmin26.
The issue was caused by other P2V that was going on and they created an issue with the SAN so than is now known.
I can get the menu for selecting a kernel to load but out of the 3 I have none boot so how would I get it into single user mode?

Thanks

Lee

[Rubberman]

If you can boot the VM from a CD, then boot it into a live/recovery CD/DVD, and from there you can run the file system utilities such as fsck to check and/or fix the VM file system.

[ltodd2]

Rubberman

I booted the redhat recovery cd and went into recovery mode. It asked if I wanted to to scan the disks and it would mount it at /mnt/sysimage. When it tried this it came back saying there were no partitions. The server was installed with LVM would that cause the error?

Thanks

[Rubberman]

Rubberman

I booted the redhat recovery cd and went into recovery mode. It asked if I wanted to to scan the disks and it would mount it at /mnt/sysimage. When it tried this it came back saying there were no partitions. The server was installed with LVM would that cause the error?

Thanks

No. I think the recovery process should recognize that it is an LVM. Maybe when the system went belly-up it trashed the partition table? There is a partition id for LVM partitions. 0x8e is the type id for an LVM, so that should be visible if you run the command "fdisk -l". If fdisk sees the partion(s) then the recovery disc cannot deal with the LVM and you will have to scan the partition manually with fsck.

[ltodd2]

I had to get sysrescure CD as the redhat cd does not have the LVM commands on it. anyways got the info about the / and /boot partitions and ran fsck on them the / had loads for errors on it. After that tried a reboot and still nothing. I loaded rescuecd again and mounted the partitions. Below is the output of /boot /boot/grub and menu.lst

When I get the grub boot menu and press a the line it has is:
<ABLE=UK crashkernel=auto rhgb quiet
Does this look ok?

/boot
dr-xr-xr-x 5 root root 1.0K Apr 18 11:44 .
drwxr-xr-x 8 root root 120 May 12 12:12 ..
-rw-r--r-- 1 root root 96K Jan 5 22:33 config-2.6.32-71.14.1.el6.x86_64
-rw-r--r-- 1 root root 96K Mar 26 20:39 config-2.6.32-71.24.1.el6.x86_64
-rw-r--r-- 1 root root 96K Sep 1 2010 config-2.6.32-71.el6.x86_64
drwxr-xr-x 3 root root 1.0K Apr 15 11:09 efi
drwxr-xr-x 2 root root 1.0K Apr 18 11:45 grub
-rw-r--r-- 1 root root 14M Apr 18 11:45 initramfs-2.6.32-71.14.1.el6.x86_64.img
-rw-r--r-- 1 root root 14M Apr 15 15:53 initramfs-2.6.32-71.24.1.el6.x86_64.img
-rw-r--r-- 1 root root 14M Apr 15 11:10 initramfs-2.6.32-71.el6.x86_64.img
drwx------ 2 root root 12K Apr 15 10:58 lost+found
-rw-r--r-- 1 root root 157K Jan 5 22:34 symvers-2.6.32-71.14.1.el6.x86_64.gz
-rw-r--r-- 1 root root 157K Mar 26 20:41 symvers-2.6.32-71.24.1.el6.x86_64.gz
-rw-r--r-- 1 root root 157K Sep 1 2010 symvers-2.6.32-71.el6.x86_64.gz
-rw-r--r-- 1 root root 2.2M Jan 5 22:33 System.map-2.6.32-71.14.1.el6.x86_64
-rw-r--r-- 1 root root 2.2M Mar 26 20:39 System.map-2.6.32-71.24.1.el6.x86_64
-rw-r--r-- 1 root root 2.2M Sep 1 2010 System.map-2.6.32-71.el6.x86_64
-rwxr-xr-x 1 root root 3.7M Jan 5 22:33 vmlinuz-2.6.32-71.14.1.el6.x86_64
-rw-r--r-- 1 root root 170 Jan 5 22:33 .vmlinuz-2.6.32-71.14.1.el6.x86_64.hmac
-rwxr-xr-x 1 root root 3.7M Mar 26 20:39 vmlinuz-2.6.32-71.24.1.el6.x86_64
-rw-r--r-- 1 root root 170 Mar 26 20:39 .vmlinuz-2.6.32-71.24.1.el6.x86_64.hmac
-rwxr-xr-x 1 root root 3.7M Sep 1 2010 vmlinuz-2.6.32-71.el6.x86_64
-rw-r--r-- 1 root root 165 Sep 1 2010 .vmlinuz-2.6.32-71.el6.x86_64.hmac


/boot/grub
root@sysresccd /mnt/backup/grub % ll
total 264K
drwxr-xr-x 2 root root 1.0K Apr 18 11:45 .
dr-xr-xr-x 5 root root 1.0K Apr 18 11:44 ..
-rw-r--r-- 1 root root 63 Apr 15 11:10 device.map
-rw-r--r-- 1 root root 13K Apr 15 11:10 e2fs_stage1_5
-rw-r--r-- 1 root root 13K Apr 15 11:10 fat_stage1_5
-rw-r--r-- 1 root root 12K Apr 15 11:10 ffs_stage1_5
-rw------- 1 root root 1.6K Apr 18 11:45 grub.conf
-rw-r--r-- 1 root root 12K Apr 15 11:10 iso9660_stage1_5
-rw-r--r-- 1 root root 13K Apr 15 11:10 jfs_stage1_5
lrwxrwxrwx 1 root root 11 Apr 15 11:10 menu.lst -> ./grub.conf
-rw-r--r-- 1 root root 12K Apr 15 11:10 minix_stage1_5
-rw-r--r-- 1 root root 14K Apr 15 11:10 reiserfs_stage1_5
-rw-r--r-- 1 root root 1.4K May 6 2010 splash.xpm.gz
-rw-r--r-- 1 root root 512 Apr 15 11:10 stage1
-rw-r--r-- 1 root root 111K Apr 15 11:10 stage2
-rw-r--r-- 1 root root 12K Apr 15 11:10 ufs2_stage1_5
-rw-r--r-- 1 root root 12K Apr 15 11:10 vstafs_stage1_5
-rw-r--r-- 1 root root 14K Apr 15 11:10 xfs_stage1_5


grub menu.lst
root@sysresccd /mnt/backup/grub % cat menu.lst
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/mapper/vg_uklft01-lv_root
# initrd /initrd-[generic-]version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux Server (2.6.32-71.14.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-71.14.1.el6.x86_64 ro root=/dev/mapper/vg_uklft01-lv_root rd_LVM_LV=vg_uklft01/lv_root rd_LVM_LV=vg_uklft01/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk crashkernel=auto rhgb quiet
initrd /initramfs-2.6.32-71.14.1.el6.x86_64.img
title Red Hat Enterprise Linux Server (2.6.32-71.24.1.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-71.24.1.el6.x86_64 ro root=/dev/mapper/vg_uklft01-lv_root rd_LVM_LV=vg_uklft01/lv_root rd_LVM_LV=vg_uklft01/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk crashkernel=auto rhgb quiet
initrd /initramfs-2.6.32-71.24.1.el6.x86_64.img
title Red Hat Enterprise Linux (2.6.32-71.el6.x86_64)
root (hd0,0)
kernel /vmlinuz-2.6.32-71.el6.x86_64 ro root=/dev/mapper/vg_uklft01-lv_root rd_LVM_LV=vg_uklft01/lv_root rd_LVM_LV=vg_uklft01/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=uk crashkernel=auto rhgb quiet
initrd /initramfs-2.6.32-71.el6.x86_64.img
root@sysresccd /mnt/backup/grub %

Any Ideas?