Volume group not found

[marcnew]

Hello, though I've been using linux (I'm from a windows background) for a server at home I'm still pretty new at it. I went with centos 5 on an amd 1400 with a raid card connected to two 1GB drives (raid1). I successfully got samba working, a media server, printer services etc. Everything was running well for quite a while. The problem came yesterday when I decided to let it upgrade the kernel using yum upgrade. (I don't really know the details I usually let it do its thing.)

So it finished doing its thing and I rebooted. I think got the error volumegroup00 not found, then it couldn't find the boot directories and finally stopped after getting a kernel panic error.

My problem was pretty much like this: /forum/redhat-fedora-linux-help/111022-fc7-volume-group-volgroup00-not-found-kernel-panic.html (add the root of this site to the beginning of this link) except it was referring to redhat kernels.

I followed the suggestion near the end, booting into rescue mode and started having some weird problem. It booted to the command line fine. I chrooted to the specified location. Now when I did an "ls" it gave me the usage prompt. So I said "ls *". Now it listed the directories on the root but it did so really strangely. Something like this

:::::::::::::
boot
:::::::::::::
shared
:::::::::::::

etc etc.

I went to the boot directory and tried to get a listing of what was in the directory. It started listing it like I showed above but after a couple files it started showing garbage... ascii charactors ...looked like what would happen if you tried to print a binary file. I tried "ls init*" it gave me one file, then it looked like was trying to print what was in the file but the first part was script then it degenerated into "binary junk".

I went ahead and backed up the img file (sorry I didn't write down the exact version so I lost it... it was something like "initrd-2.6.18-128.1.img". When I ran mkinitrd it gave me an

/etc/profile.d/lang.sh: line 76: /bin/unicode_start: cannot execute binary

along with a couple other things then errored out.

I restarted the computer and entered rescue mode. (I was hoping to get a better directory listing. Now when I tried to list the boot directory it displayed the kernel initrd-2.6.18-8.el5.img (no idea where that came from) then binary junk. I backed it up, then tried to recreate it and it gave me the same error before BUT it started going through the mkinitrd file giving errors like:

/sbin/mkinitrd: line 57: /bin/sed: cannot execute binary file

and continued with line 1030, 1048, etc etc.

I tried running it again and and now it goes through 3 of the lines above then says it can't create files in the /tmp directory because its read-only and errors out. I'm sure this is because the rescue drives are read only but I have no idea how to fix this.

My next guess is that linux needs drivers for the raid card though it never had trouble before, I never had to load anything special and I have no idea how to find and tell linux how to use them (especially in rescue mode)

So, I'm completely out of my depth, I'm not sure what to do next. In short, I'm stumped....

Any help, advice, guides, etc would be most welcome!

[MikeTbob]

Not sure what the problem here is, but you should be able to boot using the previous kernel, unless YUM removed it. I think if you can get the previous kernel booted up, you might not have these problems right now but you'll need to deal with it later, when upgrading the kernel.

[marcnew]

When I try to list the contents of the drive I get garbage so I can't even see what's out there. When I try to rebuild the kernel according the link in my previous message the mkinitrd fails horribly.

How do I get the previous kernel and install it?

[MikeTbob]

When you reboot the machine, press the spacebar to bring up the GRUB menu and select the previous kernel. Like I said, if YUM didn't remove it, you should be okay.

[marcnew]

It's that easy to boot to a previous version? so simple.

Unfortunately it's never THAT easy for me <sigh>

3 versions came up in grub:
2.6.18-128.1.10.el5
2.6.18-92.1.22.el5
2.6.18-8.e15

When I selected the second one I got these message (sans the line numbers)

1-> ext3-fs error (device dm-0): ext3_lookup: unlinked inode 197197848 in dir #197197827
2-> exec of init (/sbin/init) failed: no such file or directory
3-> Kernel panic!

I then tried booted in the oldest version and received lines 2 and 3 above.

Does this mean I have a corrupt sector on the disk? Where do I go from here? Am I hosed and need to try to compile a kernel manually from the rescue command line?

Thanks.

[MikeTbob]

You might try running fsck on this drive, it may contain errors. You might also need to re-install the kernel, it's kinda hard to say. I'd download a LiveCD for the fsck and maybe to start back up & recovery procedures.

[marcnew]

Ungh, what a pain. Hopefully it won't come down to restore. I won't have an opportunity to try this until sometime tonight.

Thanks again.

[marcnew]

Well, that took much longer than I thought to get my backup drive operating again... and back up... it was really slow.

Anyway I ran fsck on the drive and it can up with a huge amount of errors. too many to go through.

The boot directory on the drive is completely empty. It was before the fsck so whatever messed up before wasn't fixed. At least from what I can tell I don't think I lost anything else on the drive.

How do I recover the boot directory? If I run the installation disk would I be able to specify only to restore that directory? I've only installed linux by running the install disks on a "blank" machine.