Results 1 to 4 of 4
During a power outage I seem to have lost my RAID and I'm looking for guidance in recovering it. I've been running and working with RAIDS for years now but ...
- 04-29-2011 #1Just Joined!
- Join Date
- Apr 2011
- Posts
- 3
Recreate a RAID from scratch (MDADM) please help
During a power outage I seem to have lost my RAID and I'm looking for guidance in recovering it. I've been running and working with RAIDS for years now but this one had me stumped. After reading for hours I think I've the solution but I'm hesitant because there's a lot of precious, irreplaceable photo's on this server.
Setup: 11x LUKS-encrypted drives are then combined into a RAID6 using MDADM, which is then fed into a LVM setup, upon which rides the filesystem. I've fixed the LUKS portion, so the drives are decrypted, and after hours of looking at the various outputs of mdadm, I believe it's now come down to I need to manually recreate the RAID in a specific order, if you examine the output below. I know the drive SDI1 was being sync'd when the power outage hit and I'm hoping that's the drive which needs to go into the REMOVED portion...question I'd like confirmed is how do I do that? Just make sure its in the 8th spot in the creation list? Will I lose all my data doing it this way?
Thanks in advance, and I'm still open to suggestions on how to fix this horrid mess...Code:/dev/mapper/sda: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56bf - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 12 251 13 12 spare 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdd: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56af - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 5 251 5 5 active sync /dev/mapper/sdg 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sde: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b5 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 10 251 1 10 active sync /dev/mapper/sda 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdf: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b5 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 9 251 3 9 active sync /dev/mapper/sde 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdg: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56a6 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 0 251 6 0 active sync /dev/mapper/sdh 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdh: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b2 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 8 251 2 8 active sync /dev/mapper/sdd 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdi: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56be - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 11 251 14 11 spare 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdj: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b5 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 6 251 9 6 active sync /dev/mapper/sdl 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdk: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b0 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 4 251 8 4 active sync /dev/mapper/sdk 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdl: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56b0 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 11 251 18 -1 spare 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 18 -1 spare 12 12 251 13 12 spare /dev/mapper/sdm: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56a6 - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 1 251 4 1 active sync /dev/mapper/sdf 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare /dev/mapper/sdn: Magic : a92b4efc Version : 00.90.00 UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Array Size : 8790111360 (8382.90 GiB 9001.07 GB) Raid Devices : 11 Total Devices : 12 Preferred Minor : 2 Update Time : Thu Apr 28 19:36:51 2011 State : active Active Devices : 10 Working Devices : 12 Failed Devices : 1 Spare Devices : 2 Checksum : 85ab56ae - correct Events : 2868289 Chunk Size : 64K Number Major Minor RaidDevice State this 2 251 10 2 active sync /dev/mapper/sdm 0 0 251 6 0 active sync /dev/mapper/sdh 1 1 251 4 1 active sync /dev/mapper/sdf 2 2 251 10 2 active sync /dev/mapper/sdm 3 3 251 7 3 active sync /dev/mapper/sdj 4 4 251 8 4 active sync /dev/mapper/sdk 5 5 251 5 5 active sync /dev/mapper/sdg 6 6 251 9 6 active sync /dev/mapper/sdl 7 7 0 0 7 faulty removed 8 8 251 2 8 active sync /dev/mapper/sdd 9 9 251 3 9 active sync /dev/mapper/sde 10 10 251 1 10 active sync /dev/mapper/sda 11 11 251 14 11 spare 12 12 251 13 12 spare
- 04-29-2011 #2Linux Enthusiast
- Join Date
- Oct 2005
- Location
- Italy
- Posts
- 718
Do you have plugged in a replacement drive? What's the output of:
# cat /proc/mdstat
To me it would look like /dev/mapper/sda and /dev/mapper/sdi are spare? Why not use one of those and plug in another one and let it become spare?Last edited by Kloschüssel; 04-29-2011 at 11:34 AM.
- 04-29-2011 #3Just Joined!
- Join Date
- Apr 2011
- Posts
- 3
Thanks for the reply,
SDA is indeed a spare, SDI is the drive that was being sync'd but now shows up as a spare.Code:cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md2 : inactive dm-1[12](S) dm-4[9](S) dm-8[4](S) dm-7[6](S) dm-3[10](S) dm-5[0](S) dm-11[2](S) dm-2[5](S) dm-6[8](S) dm-10[1](S) dm-12[11](S) 10743629056 blocks
Trying to force the raid to run:
and finally:Code:mdadm --run --force /dev/md2 mdadm: failed to run array /dev/md2: Input/output error
SDI should be drive #7, and I think if I can get it back into slot 7 it'll jumpstart the RAID.Code:mdadm -D /dev/md2 /dev/md2: Version : 00.90 Creation Time : Thu Sep 9 09:05:18 2010 Raid Level : raid6 Used Dev Size : 976679040 (931.43 GiB 1000.12 GB) Raid Devices : 11 Total Devices : 11 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Thu Apr 28 19:36:51 2011 State : active, degraded, Not Started Active Devices : 9 Working Devices : 11 Failed Devices : 0 Spare Devices : 2 Chunk Size : 64K UUID : de60075b:0cfb4ec0:df8d1179:3cf1071f (local to host zaphod) Events : 0.2868289 Number Major Minor RaidDevice State 0 251 5 0 active sync /dev/mapper/sdg 1 251 10 1 active sync /dev/mapper/sdm 2 251 11 2 active sync /dev/mapper/sdn 3 0 0 3 removed 4 251 8 4 active sync /dev/mapper/sdk 5 251 2 5 active sync /dev/mapper/sdd 6 251 7 6 active sync /dev/mapper/sdj 7 0 0 7 removed 8 251 6 8 active sync /dev/mapper/sdh 9 251 4 9 active sync /dev/mapper/sdf 10 251 3 10 active sync /dev/mapper/sde 11 251 12 - spare /dev/mapper/sdi 12 251 1 - spare /dev/mapper/sda
- 04-30-2011 #4Just Joined!
- Join Date
- Apr 2011
- Posts
- 3
Does the fact that the drives individually think they're active (in mdadm -E /dev/mapper/sd[x]) meanwhile the raid itself thinks it's in degraded/diry (mdadm -D /dev/md2) ?
Reply With Quote 
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts