RHEL 4 AS kernel panic... need urgent help

[neoscreenager]

Hi
I have RHEL 4 AS running on HP ProLiant DL580 G5. The machine was running fine last week but was shutted down abruptly due to power and ups faliure. When i booted it again i got following error :

Redhat Nash Version 4.2.1.8
can't access (null)
exec of init ((null)) failed!!! : 14
umount /initrd/dev failed :2
Kernel panic -not syncing! Attempted to kill init!!

When i booted from the RHEL 4 CD#1 into the rescue mode, it lodes the cciss drivers and ask for whether to mount /mnt/sysimage or directly dropped to the shell. I choose continue so that /mnt/sysimage could be mounted but it shows an error that no linux partitions found , dropping to the shell.

At the shell when i did fdisk -l , it gives following output :

Disk /dev/cciss/c0d0: 146.7 GB, 146745262080 bytes
255 heads, 63 sectors/track, 17840 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000ba26d

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 127 1020096 83 Linux
/dev/cciss/c0d0p2 128 7775 61432560 83 Linux
/dev/cciss/c0d0p3 7776 13894 49150867+ 82 Linux swap / Solaris
/dev/cciss/c0d0p4 13895 17840 31696245 f W95 Ext'd (LBA)
/dev/cciss/c0d0p5 13895 17718 30716248+ 83 Linux

Here c0d0p1 is /boot, c0d0p2 is / and c0d0p5 is /opt.

I tried to mount all the partitions and they are mounting fine but the / partition (c0d0p2) does not have /bin and /etc folders, insted of which there are files of same name.

when i tried mkinitrd, it says command not found.

I am not very litrate about RAID . Please help me in correcting this problem, its very urgent coz we have a demo next monday.

Here is my grub.conf file::

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/cciss/c0d0p2
# initrd /initrd-version.img
boot=/dev/cciss/c0d0
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title Red Hat Enterprise Linux AS (2.6.9-42.ELsmp)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.ELsmp ro root=LABEL=/ rhgb quiet
initrd /initrd-2.6.9-42.ELsmp.img
title Red Hat Enterprise Linux AS-up (2.6.9-42.EL)
root (hd0,0)
kernel /vmlinuz-2.6.9-42.EL ro root=LABEL=/ rhgb quiet
initrd /initrd-2.6.9-42.EL.img


To make things worse, I booted from fedora 9 cd in rescue mode and tried following commands:

device (hd0) /dev/cciss/c0d0
root (hd0,0)
setup (hd0)

Now even the grub is not loading, it just displays Grub Error without any error number

I've ran fsck on /dev/cciss/c0d0p1, p2 and p5, but nothing..


HELP!!!!

Thanks

[Lakshmipathi]

I tried to mount all the partitions and they are mounting fine

that's a good news...you can backup important data.

but the / partition (c0d0p2) does not have /bin and /etc folders, insted of which there are files of same name

are you sure / don't have directories like /bin and /etc???
If possible login and another RHEL 4 AS and mount your older c0d0p2 into it.
Then copy /bin content from this RHEL 4 to problematic system ..now you have /bin files.

But i guess - it's not easy to just copy /etc into c0d0p2 because configurations will be different for these two machines.
Tough choice would be,copy /etc directory too into c0d0p2 and reboot the system with original RHEL and the start debugging according to what system says.

You would needed edit files like /etc/fstab

Instead of doing above said experimental work, first drop mail in linux developer mailing list or linux users list or ext3 users list with your problem ..if files /etc/ and /bin are deleted for sure ,and have not reboot since then.You could try
"ext3grep" it retrieves files from journal.

[neoscreenager]

Thanks Lakshmi for the reply, but here is a catch.. ext3grep only recovers the deleted files and not repair the corrupted files... my /bin and /etc folders are not deleted but are corrupted i.e when i ls into my root directory, i can see bin and etc but as files (not as folder ). Is there any way to repair the etc and bin to convert them back to normal /etc and /bin folders with all the configuration files intact..
I hope you don't mind if i ask you to please provide me name of some linux developer mailing list or linux users list or ext3 users lists...sorry for bothering..

Thanks a lot...

[Lakshmipathi]

my /bin and /etc folders are not deleted but are corrupted i.e when i ls into my root directory, i can see bin and etc but as files (not as folder ).

Can you post the output of

ls -l /

Is there any way to repair the etc and bin to convert them back to normal /etc and /bin folders with all the configuration files intact..

You said you have ran fsck- that didn't fix this ?

linux developer mailing list or linux users list or ext3 users lists...

It looks like ext3 file system issue...so here is the link.
https://www.redhat.com/mailman/listinfo/ext3-users

sorry for bothering..

No..not at all - you are most welcome

RHEL 4 is commerical version of Redhat , If you have proper licence that didn't expired yet , you can contact them.

[neoscreenager]

here is the ls -l of my root directory:::

total 728
-rwxr-xr-x 1 root root 88468 Feb 10 2006 bin
drwxr-xr-x 4 root root 4096 Mar 11 2008 boot
drwxr-xr-x 2 root root 4096 Mar 11 2008 dev
-rwxr-xr-x 1 root root 464080 Jul 4 2006 etc
-rw-r--r-- 1 root root 0 May 27 02:21 halt
drwxr-xr-x 19 root root 4096 May 8 06:38 home
drwxr-xr-x 2 root root 4096 Aug 12 2004 initrd
drwx------ 86 root root 16384 Mar 12 2008 lost+found
drwxr-xr-x 3 root root 4096 May 27 00:34 media
drwxr-xr-x 2 root root 4096 Jul 11 2006 misc
drwxr-xr-x 2 root root 4096 Aug 12 2004 mnt
drwxr-xr-x 10 root root 4096 May 4 01:11 opt
drwxr-xr-x 2 root root 4096 Mar 12 2008 proc
-rwxr-xr-x 1 root root 1484 Dec 5 2005 root
lrwxrwxrwx 1 root root 19 Mar 11 2008 sbin -> ../init.d/haldaemon
drwxr-xr-x 2 root root 4096 Mar 11 2008 selinux
drwxr-xr-x 2 root root 4096 Aug 12 2004 srv
drwxr-xr-x 2 root root 4096 Mar 12 2008 sys
drwxr-xr-x 3 root root 4096 Mar 11 2008 tftpboot
drwxrwxrwt 44 root root 16384 May 27 02:21 tmp
drwxr-xr-x 2 root root 4096 Jul 21 2008 user
drwxr-xr-x 17 root root 4096 Sep 25 2008 usr
drwxr-xr-x 24 root root 4096 Mar 13 2008 vars

i have run fsck but it did'nt fix anythng...

[Lakshmipathi]

Yes,bin,etc and root all seems like regular file but strangely date seems be retains it's original value.. Below soultion is bit risky..check with ext3-users before proceeding with it (make sure you backup important data)...I guess you could use commands like debugfs and modify the file type of bin to directory and see whether that rectifies the problem.

[neoscreenager]

hey laxmi,
i have a db2server of exact same configuration, even the same kinda partitions, so i copied /etc/ and /bin from this machine to the faulty one. When i copied the files, all files except the symbolic links get copied , so i made each link by hand, in /bin, i have made a link "sh" to the bash, then i rebooted it with the RHEL 4 AS CD #1 into rescue mode. This time it detected the linux file system and successfully mounted /mnt/sysimage. But when i tried to chroot , it says /bin/sh command not found.
I rebooted the machine without the CD, it started to boot and stopped at this message:

exec of init (/bin/init) failed :2
umount /initrd/dev failed :2
Kernel panic -not syncing! Attempted to kill init!!

I know copying /bin,/etc is very stupid, but i thought it might work as the configurations are almost the same, i don't know why it is giving /bin/sh not found error when sh is right there in /bin, though its a link to bash.
Since the rescue disk is recognizing the linux partitions now so i tried to run update linux from it, it ran normally until it asked for disk 4, install a package sucessfully from it and give error while installing the next package. It say : its a fatal error, either the media is corrupt or the there is no enough disk space, click ok to reboot.
I checked the media, and the space, both are good.
What should i do now?

[Lakshmipathi]

Before copying /etc/ and /bin ,what's your file system size? Does it decreased
because of bin and etc "changed" as a regular file?

Check hints from /var/log/messages file.Now you copied new /etc/ and /bin what happened to older bin and etc ..have you renamed it or deleted them?

If you File system size didn't decreased even though bin and etc shown as regular file , then if you still have those files. Please first try with debugfs with option mi and modify the inode mode value (say as [040755] 04 - denotes directory type )

example:

debugfs /dev/sda4 -w

mi /bin
mode [something_here] 040755

and just press enter for other fields

do the same for /etc

and reboot the system

------------
Regarding your current error , I would prefer using rescue + chroot to resolve this issue.
try whether

strace chroot /dev/your_device

throws any specific details about /bin/sh

[neoscreenager]

hey laxmi,
i have a db2server of exact same configuration, even the same kinda partitions, so i copied /etc/ and /bin from this machine to the faulty one. When i copied the files, all files except the symbolic links get copied , so i made each link by hand, in /bin, i have made a link "sh" to the bash, then i rebooted it with the RHEL 4 AS CD #1 into rescue mode. This time it detected the linux file system and successfully mounted /mnt/sysimage. But when i tried to chroot , it says /bin/sh command not found.
I rebooted the machine without the CD, it started to boot and stopped at this message:

exec of init (/bin/init) failed :2
umount /initrd/dev failed :2
Kernel panic -not syncing! Attempted to kill init!!

I know copying /bin,/etc is very stupid, but i thought it might work as the configurations are almost the same, i don't know why it is giving /bin/sh not found error when sh is right there in /bin, though its a link to bash.
Since the rescue disk is recognizing the linux partitions now so i tried to run update linux from it, it ran normally until it asked for disk 4, install a package sucessfully from it and give error while installing the next package. It say : its a fatal error, either the media is corrupt or the there is no enough disk space, click ok to reboot.
I checked the media, and the space, both are good.
What should i do now?

[Lakshmipathi]

please check by reply above