Boot hangs after SELinux: Disabled at runtime

[DaveJL]

Hello,

My Redhat box hangs on boot up after "SELinux: Disbled at runtime" ...

raid1: raid set md0 active with 2 out of 2 mirrors
md: ... autorun DONE.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
SELinux:  Disabled at runtime.
type=1404 audit(1268063367.600:2): selinux=0 auid=4294967295 ses=4294967295

This boot hanging problem started after I ran /etc/rc.d/rc.sysinit to reset the hostname, but the power went out before the script completed.

I'm suspecting that the hard kill messed up some boot files, and I'm hoping a kind & wise soul can help guide me on how to restore them properly.

Thanks very much.
Dave

[devils casper]

Which version of RedHat are you using? Did you change anything in /etc/fstab file? Boot up from LiveCD of any Linux distro and check the contents of /etc/fstab file.

[DaveJL]

Thanks devils casper,

No, I didn't touch anything in /etc/fstab. Could it have been corrupted when the power-outage occurred during the /etc/rc.d/rc.sysinit process?

After about 2 hours, the boot output pushed an inch forward to include "RAID1 conf printout":

SELinux:  Disabled at runtime.
type=1404 audit(1268063367.600:2): selinux=0 auid=4294967295 ses=4294967295
RAID1 conf printout:
 --- wd:2 rd:2
 disk 0, wo:0, o:1, dev:sda3
 disk 1, wo:0, o:1, dev:sdb3

But, like before, it's hanging here again (already about 2 hours and counting).

Should I just let it continue and do whatever it's doing?

Thanks again,
Dave

[devils casper]

2 Hours is too much time. How much disk space do you have?

[DaveJL]

The HD size is 500 GB.

[devils casper]

Let it update settings for a while. In case it takes too much time, boot up from LiveCD or try booting in Rescue Mode and check logs for relevant messages/info.

[DaveJL]

Ok, will do, thanks.

And just in case I need to go hunting in Rescue Mode, which logs should I examine for the relavent messages/info?

I'm assuming somewhere in /var/logs/...

[DaveJL]

I'm now in Recovery Mode.

My /etc/fstab :

/dev/md1                /                       ext3    defaults,usrquota        1 1
/dev/md0                /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sdb2         swap                    swap    defaults        0 0
LABEL=SWAP-sda2         swap                    swap    defaults        0 0

fdisk -l yields:

# fdisk -l

Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0001c6cf

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14         655     5156865   82  Linux swap / Solaris
/dev/sda3             656       60801   483122745   fd  Linux raid autodetect

Disk /dev/sdb: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0002ec98

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   fd  Linux raid autodetect
/dev/sdb2              14         655     5156865   82  Linux swap / Solaris
/dev/sdb3             656       60801   483122745   fd  Linux raid autodetect

Disk /dev/md1: 494.7 GB, 494717566976 bytes
2 heads, 4 sectors/track, 120780656 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Disk identifier: 0x00000000

Disk /dev/md1 doesn't contain a valid partition table

mdadm -E -s shows:

# mdadm -E -s
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=629bfb37:53a8883f:537f768e:e9290991
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=adc78fa5:fac6da39:5ea2599b:29ca6912

When I try to mount /dev/md0, I get the error:

# mount -t ext3 /dev/md0 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or helper program, or other error
       (could this be the IDE device where you in fact use
       ide-scsi so that sr0 or sda or so is needed?)
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

dmesg | tail shows:

# dmesg | tail
[ 1349.012161] unionfs: new lower inode mtime (bindex=0, name=mdadm)
[ 1356.803854] md: md1 still in use.
[ 1356.870311] md: bind<sdb3>
[ 1356.906846] raid1: raid set md1 active with 1 out of 2 mirrors
[ 1356.906867] md1: detected capacity change from 0 to 494717566976
[ 1356.906978]  md1: unknown partition table
[ 1389.504015] EXT4-fs (md0): unable to read superblock
[ 1398.655582] EXT3-fs: unable to read superblock
[ 1808.394513] EXT2-fs: unable to read superblock
[ 1836.473091] EXT3-fs: unable to read superblock

When I try to fsck /dev/md0, I get:

# fsck /dev/md0
fsck 1.40.8 (13-Mar-2008)
e2fsck 1.40.8 (13-Mar-2008)
fsck.ext2: Invalid argument while trying to open /dev/md0

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I'm not sure how to proceed ... any guidance would be much appreciated.

Thanks,
Dave