problems with raid1/0 UUID

[zolfo]

Hi everybody,
I'm kinda copy/pasting this problem in different forums to get somewhere...
I have 4 disks managed by mdadm in a proprietary storage, 2 mirrors with a raid0 on top of them running from 2009.
The problems began when all four disks reported smart errors and the storage web interface asked me to recreate the raid from scratch to make it work (destroying all the data), so I took out the disks and mount them on my linux machine to see if the data where still there. Easy with mdadm I reassembled the 2 raid1 and the raid0, activated the LVM and mount it. All data where there. Yesterday I did the same thing with a new hd plugged to copy the data to it but less luckily since 1 disk is dying badly, and after a lot of commands to make it work I screwed it, now one disk reports a wrong UUID and two disks as spare. Here is the output of mdadm -E of the 4 disks:

/dev/sda1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c65b7bc7:9c0f7ea7:c218b028:629824f1
  Creation Time : Wed Dec  2 14:43:23 2009
     Raid Level : raid1
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Tue Sep  4 05:51:40 2012
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1
       Checksum : 5dca950 - correct
         Events : 124877


      Number   Major   Minor   RaidDevice State
this     2       8      113        2      spare   /dev/sdh1

   0     0       0        0        0      removed
   1     1       8       33        1      active sync   /dev/sdc1
   2     2       8      113        2      spare   /dev/sdh1


/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 4b81e869:37272dde:2ecf36a3:b99d24d5
  Creation Time : Wed Dec  2 14:43:23 2009
     Raid Level : raid1
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Tue Sep  4 05:46:38 2012
          State : active
 Active Devices : 1
Working Devices : 2
 Failed Devices : 1
  Spare Devices : 1
       Checksum : e9d8ae7e - correct
         Events : 203312


      Number   Major   Minor   RaidDevice State
this     2       8        1        2      spare   /dev/sda1

   0     0       8       97        0      active sync   /dev/sdg1
   1     1       0        0        1      faulty removed
   2     2       8        1        2      spare   /dev/sda1


/dev/sdc1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c65b7bc7:9c0f7ea7:c218b028:629824f1
  Creation Time : Wed Dec  2 14:43:23 2009
     Raid Level : raid1
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 1

    Update Time : Sat Oct  8 05:45:16 2011
          State : clean
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1
       Checksum : 42333d0 - correct
         Events : 1498


      Number   Major   Minor   RaidDevice State
this     2     254        2        2      spare

   0     0       0        0        0      removed
   1     1     254        3        1      active sync
   2     2     254        2        2      spare



/dev/sdd1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : c65b7bc7:9c0f7ea7:c218b028:629824f1
  Creation Time : Wed Dec  2 14:43:23 2009
     Raid Level : raid1
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 0

    Update Time : Tue Sep  4 07:47:51 2012
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 5dcc3a6 - correct
         Events : 124881


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       0        0        0      removed
   1     1       8       17        1      active sync   /dev/sdb1

/dev/sdd1 is reporting the wrong UUID as it should be like /dev/sdc1, I think that I did wrong removing it from the correct array and readding it to the wrong one, now it's seen as a spare disk like /dev/sdc1, which are not as they are both part of the same mirror and there should not be any spare drive. The main problem is if I try to force assemble the array with the single disks it fails with error:

mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.

and from dmesg:

raid1: no operational mirrors for md0

the only one that is working is /dev/sdd1 that starts correctly and shows the raid0 part:

mdadm --assemble /dev/md0 /dev/sdd1 --force --run
mdadm: /dev/md0 has been started with 1 drive (out of 2).
# mdadm -E /dev/md0
/dev/md0:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : d5d800c7:432da000:632ad3a3:473452ce
  Creation Time : Wed Dec  2 14:43:23 2009
     Raid Level : raid0
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 2

    Update Time : Wed Dec  2 14:43:23 2009
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 3cf33646 - correct
         Events : 2

     Chunk Size : 32K

      Number   Major   Minor   RaidDevice State
this     1       9        1        1      active sync

   0     0       9        0        0      active sync   /dev/md0
   1     1       9        1        1      active sync

I'd like to be able to force start the array from /dev/sdb1 with UUID 4b81e869:37272dde:2ecf36a3:b99d24d5 so I can assemble the raid0 with all my data,
but it keeps telling me Input/Output error, what can I do?

[atescomp]

Yeah, it is hosed. Generally, UUIDs are never "wrong". UUIDs are assigned when the disk volumes/partitions are created. The disks must have been reinitialized at some point in all the confusion since they were assigned new UUIDs. It must have seen the disks as "new" or unformatted at some point if it autoassigned them as spares.