Find the answer to your Linux question:
Results 1 to 4 of 4
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    Recreate a RAID from scratch (MDADM) please help


    During a power outage I seem to have lost my RAID and I'm looking for guidance in recovering it. I've been running and working with RAIDS for years now but this one had me stumped. After reading for hours I think I've the solution but I'm hesitant because there's a lot of precious, irreplaceable photo's on this server.

    Setup: 11x LUKS-encrypted drives are then combined into a RAID6 using MDADM, which is then fed into a LVM setup, upon which rides the filesystem. I've fixed the LUKS portion, so the drives are decrypted, and after hours of looking at the various outputs of mdadm, I believe it's now come down to I need to manually recreate the RAID in a specific order, if you examine the output below. I know the drive SDI1 was being sync'd when the power outage hit and I'm hoping that's the drive which needs to go into the REMOVED portion...question I'd like confirmed is how do I do that? Just make sure its in the 8th spot in the creation list? Will I lose all my data doing it this way?


    Code:
    /dev/mapper/sda:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56bf - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this    12     251       13       12      spare
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdd:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56af - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     5     251        5        5      active sync   /dev/mapper/sdg
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sde:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b5 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this    10     251        1       10      active sync   /dev/mapper/sda
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdf:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b5 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     9     251        3        9      active sync   /dev/mapper/sde
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdg:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56a6 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     0     251        6        0      active sync   /dev/mapper/sdh
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdh:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b2 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     8     251        2        8      active sync   /dev/mapper/sdd
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdi:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56be - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this    11     251       14       11      spare
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdj:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b5 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     6     251        9        6      active sync   /dev/mapper/sdl
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdk:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b0 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     4     251        8        4      active sync   /dev/mapper/sdk
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdl:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56b0 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this    11     251       18       -1      spare
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       18       -1      spare
      12    12     251       13       12      spare
    /dev/mapper/sdm:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56a6 - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     1     251        4        1      active sync   /dev/mapper/sdf
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    /dev/mapper/sdn:
              Magic : a92b4efc
            Version : 00.90.00
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
         Array Size : 8790111360 (8382.90 GiB 9001.07 GB)
       Raid Devices : 11
      Total Devices : 12
    Preferred Minor : 2
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active
     Active Devices : 10
    Working Devices : 12
     Failed Devices : 1
      Spare Devices : 2
           Checksum : 85ab56ae - correct
             Events : 2868289
    
         Chunk Size : 64K
    
          Number   Major   Minor   RaidDevice State
    this     2     251       10        2      active sync   /dev/mapper/sdm
    
       0     0     251        6        0      active sync   /dev/mapper/sdh
       1     1     251        4        1      active sync   /dev/mapper/sdf
       2     2     251       10        2      active sync   /dev/mapper/sdm
       3     3     251        7        3      active sync   /dev/mapper/sdj
       4     4     251        8        4      active sync   /dev/mapper/sdk
       5     5     251        5        5      active sync   /dev/mapper/sdg
       6     6     251        9        6      active sync   /dev/mapper/sdl
       7     7       0        0        7      faulty removed
       8     8     251        2        8      active sync   /dev/mapper/sdd
       9     9     251        3        9      active sync   /dev/mapper/sde
      10    10     251        1       10      active sync   /dev/mapper/sda
      11    11     251       14       11      spare
      12    12     251       13       12      spare
    Thanks in advance, and I'm still open to suggestions on how to fix this horrid mess...

  2. #2
    Linux Engineer Kloschüssel's Avatar
    Join Date
    Oct 2005
    Location
    Italy
    Posts
    773
    Do you have plugged in a replacement drive? What's the output of:

    # cat /proc/mdstat

    To me it would look like /dev/mapper/sda and /dev/mapper/sdi are spare? Why not use one of those and plug in another one and let it become spare?
    Last edited by Kloschüssel; 04-29-2011 at 11:34 AM.

  3. #3
    Thanks for the reply,

    Code:
    cat /proc/mdstat 
    Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] 
    md2 : inactive dm-1[12](S) dm-4[9](S) dm-8[4](S) dm-7[6](S) dm-3[10](S) dm-5[0](S) dm-11[2](S) dm-2[5](S) dm-6[8](S) dm-10[1](S) dm-12[11](S)
          10743629056 blocks
    SDA is indeed a spare, SDI is the drive that was being sync'd but now shows up as a spare.

    Trying to force the raid to run:
    Code:
    mdadm --run --force /dev/md2
    mdadm: failed to run array /dev/md2: Input/output error
    and finally:
    Code:
    mdadm -D /dev/md2
    /dev/md2:
            Version : 00.90
      Creation Time : Thu Sep  9 09:05:18 2010
         Raid Level : raid6
      Used Dev Size : 976679040 (931.43 GiB 1000.12 GB)
       Raid Devices : 11
      Total Devices : 11
    Preferred Minor : 2
        Persistence : Superblock is persistent
    
        Update Time : Thu Apr 28 19:36:51 2011
              State : active, degraded, Not Started
     Active Devices : 9
    Working Devices : 11
     Failed Devices : 0
      Spare Devices : 2
    
         Chunk Size : 64K
    
               UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod)
             Events : 0.2868289
    
        Number   Major   Minor   RaidDevice State
           0     251        5        0      active sync   /dev/mapper/sdg
           1     251       10        1      active sync   /dev/mapper/sdm
           2     251       11        2      active sync   /dev/mapper/sdn
           3       0        0        3      removed
           4     251        8        4      active sync   /dev/mapper/sdk
           5     251        2        5      active sync   /dev/mapper/sdd
           6     251        7        6      active sync   /dev/mapper/sdj
           7       0        0        7      removed
           8     251        6        8      active sync   /dev/mapper/sdh
           9     251        4        9      active sync   /dev/mapper/sdf
          10     251        3       10      active sync   /dev/mapper/sde
    
          11     251       12        -      spare   /dev/mapper/sdi
          12     251        1        -      spare   /dev/mapper/sda
    SDI should be drive #7, and I think if I can get it back into slot 7 it'll jumpstart the RAID.

  4. $spacer_open
    $spacer_close
  5. #4
    Does the fact that the drives individually think they're active (in mdadm -E /dev/mapper/sd[x]) meanwhile the raid itself thinks it's in degraded/diry (mdadm -D /dev/md2) ?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •