Linux does not see sdc on boot (but does during install)

OK, now I'm at a loss.

I have been struggling with getting my Ubuntu server to boot. After a couple of days of fighting with it, I gave up and reinstalled thinking that'll sort it out.

It didn't.

What I get is a kernel panic (the same as when I tried to fix things) saying that boot can't find sdc2 (where root is located, sdc1 is /boot). It only offers me the partitions for sda and sdb drives, but not sdc. But here's the kicker: I just installed the system and I definitely could see sdc!

Reason I wouldn't want to mess up the partition table is that I have already set it up (and I have a RAID5 setup as well that I don't want to accidentally mess up).

How is this even possible? It does look like the order of the disks has somehow changed (because I'm pretty sure I had boot and root on sda - although I'm not 100% sure but if not, I don't know what I was thinking). Trying to change the boot order in BIOS did nothing, I get exactly the same setup during install.

EDIT. A couple of more specific questions:
1. What determines the order of the devices in Linux? I.e. /dev/sda, /dev/sdb etc. as well as (hd0,gpt1), (hd1,gpt1) etc? gpt parts is easy enough (order they are on the disk), but it seems these can even be different (sda == hd1 and sdb == hd0).
2. Any idea why kernel doesn't see sdc but sees the other two devices?
3. Do I need the bios_grub flag on one of the partitions? System does attempt to boot without it.

I have pictures of the boot error, /etc/fstab and blkid, but seems I'm not allowed to post links to other sites.

I have tried to reinstall GRUB, edit grub.cfg directly, change parameters in boot menu ('e' in grub menu), reinstall the whole system, change boot order in BIOS menu and.. I can't even remember what I have tried (my initial attempt was a couple of weeks ago).

Before reinstall /etc/fstab said boot and root were on sda rather than sdc, but something must have messed up the order the disks are reported. But even in this case, why does this not work? I thought I could organize the partitions as I want.

Now I'm running out of ideas. Do I go and find 4 terabytes of backup storage somewhere, copy the RAID5 data there and wipe the whole system clean? Or just change boot and root to sda and home to sdc and re-format every partition apart from the RAID partitions (I didn't re-format during the re-installation)?

I'm also slightly worried how to rebuild the RAID array, since it didn't occure to me to backup mdamd.conf before re-installing the system.

Boot from a live CD/DVD/USB drive and look at the drives. Assuming the file systems are intact, you can then label them and edit the /etc/fstab file on the root partition, changing the devices from using UUID's or /dev/sdxN identifiers to using LABEL=name. The reason for this is if it is using /dev/sdxN to identify the volume, that can change on boot, and what was sda, sdb, or sdc on installation may no longer be that when you reboot, depending upon the order the BIOS "discovers" them. Using labels will eliminate that altogether - I've had this problem myself in the past. FWIW, here is an example of my /etc/fstab:

Code:

UUID=0e9c3025-59d3-424e-818f-c305c819b4e7 / ext4 defaults 1 1 UUID=7819b966-a8e6-4d61-87c1-d377e3074bd8 /boot ext4 defaults 1 2 LABEL=swap1 swap swap defaults 0 0 /dev/mapper/VolGroup00-LogVol00 /home ext3 defaults 1 2 # LABEL=afs-esata-a1 /mnt/array1 ext4 defaults 1 2 LABEL=afs-ts02 /mnt/ts02 ext4 defaults 1 2 LABEL=afs-esata-1 /mnt/esata1 ext4 defaults 0 0 LABEL=afs-esata-2 /mnt/esata2 ext4 defaults 0 0 LABEL=afs-esata-3 /mnt/esata3 ext4 defaults 0 0 LABEL=afs-esata-4 /mnt/esata4 ext4 defaults 0 0 # LABEL=afs-esata-5 /mnt/esata5 ext4 defaults 0 0 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0

Note that / and /boot are using UUID's. If I need to replace those drives and clone them, the UUID will change, so in that case I will have to label them like the others and use the LABEL=name format. IE, I would change it to something like this:

Code:

LABEL=_/ / ext4 defaults 1 1 LABEL=_/boot /boot ext4 defaults 1 2 LABEL=swap1 swap swap defaults 0 0 /dev/mapper/VolGroup00-LogVol00 /home ext3 defaults 1 2 . . .

I have looked at the drives but unfortunately it seems impossible to link to the images that shows them here. I can't even link to a page that has links...

Anyways, /etc/fstab has lines like

UUID=90b.... / ext3 (etc)
UUID=06c... /boot ext3 (etc)

blkid shows that boot is /dev/sdc1 and root is /dev/sdc2

Thing is those two UUID's are not even shown in the kernel error as possible partitions to choose as root (I have a picture of the kernel error, but again, I can't share it). I have booted the machine a number of times and every time I get the same drives visible to kernel, no matter what boot order I define in BIOS (does this even matter?).

So I gather I can use the label as the root parameter in boot?

You should definitely try labels. Unless you are dual-booting the system and some other OS is on /dev/sda etc. then it should not see the root/boot partitions as being on /dev/sdc - normally they would be /dev/sda. So, some other drive(s) is/are definitely getting recognized first. Since these are apparently sata drives, have you tried changing the port they are plugged into?

No I haven't. That might actually work and is pretty straightforward since the kernel error shows which partitions it detects. I could see if it helps straight away.

ARGH! I had a long message written and accidentally pressed "Reply to Thread".

Anyways, to cut a long story short, it looks like there was an issue with my mobo. I had the boot/root disk in SATA2_4 port and looks like that doesn't work properly. Now I have the disks set so that I have boot/root in SATA2_0, and it shows up as sda with boot being sda1 and root as sda2.

Trouble is, I can't get the grub parameters set up correctly. I have them as

set root='(hd0,gpt1)'
search --no-floppy --fs-uuid --setroot 7149a010-7e61-4d70-9a28-3f637978bef8
linux /vmlinuz-3.2.0-33-generic root=/dev/sda2 ro

I haven't managed to find any explanations to those parameters but I suspect I should have the boot partition somewhere in there in one of the roots? That UUID is the root partitions UUID.

I just get "no such device <UUID>" even though I'm positive it's correct.

I reinstalled once more with the cables set in better SATA ports and the system is now up and running. Thanks all for your help!