Mdadm problems.....i'm dying please help

**wonslung** · 04-03-2009

ok....last night i had a power outage which caused one of my drives to drop out of the array....i have a NAS box with 6 1TB hard drives in software raid 5
i've had drives drop out of the array before because of issues with the case i have(it's got 6 hot swap bays for the drives and the original ones were loose...anyways, that's been taken care of) and i've also added new drives to the array (i started with 3) so i didn't think it was a big deal....when i woke up today i found that the array was down....4 drives up, one failed and one spare (it's supposed to be 6, somehow the one didn't make it back in and another one failed before it finished....)

well i thought this was the end.....but just to be sure i did some research....the even numbers were the same on 4 drives and the 5th was only off by 7, i was able to bring the array back up with a

Code:

mdadm --assemble --force /dev/md0

it comes up with 5 drives and one spare, so it starts to rebuild..i can mount it...read data and all that and it seems to be fine but when it gets to 100% it fails again UUUU__

i'm at a total loss.....i really don't have the extra space to move all the data and rebuild the array.....besides....i'm not even sure what's going on to be honest...i'm kind of a newb

i'll post as much info as i can think to post below....if you need anything else to help me, let me know....i'm going nuts here....please help

if i just try to assemble it, it won't assemble

Code:

# mdadm -A --scan
mdadm: /dev/md0 assembled from 4 drives and 1 spare - not enough to start the array.

but if i force it, it does

Code:

root@wonslung-raid:/home/wonslung# mdadm -Af --scan
mdadm: forcing event count in /dev/sdc1(5) from 1696710 upto 1696718
mdadm: clearing FAULTY flag for device 3 in /dev/md0 for /dev/sdc1
mdadm: /dev/md0 has been started with 5 drives (out of 6) and 1 spare.

i can then mount it and browse and use the files

Code:

root@wonslung-raid:/home/wonslung# mount /dev/md0 /mnt/raid/
root@wonslung-raid:/home/wonslung# cd /mnt/raid
root@wonslung-raid:/mnt/raid# ls
nas  pfsense  tomato  wonslung
root@wonslung-raid:/mnt/raid# cd nas
root@wonslung-raid:/mnt/raid/nas# ls
config-pfsense.mshome-20090327003027.xml
dmesg.1.gz
dmesg.2.gz
Laptop_A205-S4639.7z
Net2FTP
NVIDIA-Linux-x86-180.22-pkg1.run
NVIDIA-Linux-x86-180.29-pkg1.run
pfSense-2.0-ALPHA-ALPHA-20090326-0414.iso.gz
pfsense backups
phpwebftp
profileFx3{default}.fbu
putty.exe
torrents
Video
VirtualBox-2.1.4-43001-Linux_x86.run
vlcmod
root@wonslung-raid:/mnt/raid/nas# cd Video/
root@wonslung-raid:/mnt/raid/nas/Video# ls
Highres  Movies  Television  Three Stooges
root@wonslung-raid:/mnt/raid/nas/Video# cd Television/
root@wonslung-raid:/mnt/raid/nas/Video/Television# ls
24                                Life
30.Rock                           Life.on.Mars
Afro.Samurai                      Lost
Alias                             Meadowlands
Battlestar.Galactica              Mr.Show
Beavis.and.Butthead               My.Name.is.Earl
Big.Love                          My.Own.Worst.Enemy
Bones                             Nip.Tuck
Boomtown                          Numb3rs
Breaking.Bad                      Other
Brotherhood                       Prison.Break
Californication                   Pushing.Daisies
Chuck                             Reaper
Crash                             Rescue.Me
Crusoe                            Robot.Chicken
Curb.Your.Enthusiasm              Scrubs
Da.Ali.G.Show                     Seinfeld
Dark Angel                        Sopranos
Dead Like Me                      South.Park
Deadwood                          Superjail
Dexter                            Terminator.The.Sarah.Connor.Chronicles
ER                                Testees
Flight.of.the.Conchords           The.Closer
Freaks & Geeks                    The.Office
Fringe                            The Riches
Grey's.Anatomy                    The Universe
Heroes                            The.Wire
House                             Undeclared
Its.Always.Sunny.In.Philadelphia  United.States.of.Tara
Lie.to.Me                         Weeds
root@wonslung-raid:/mnt/raid/nas/Video/Television# cd Sopranos/
root@wonslung-raid:/mnt/raid/nas/Video/Television/Sopranos# ls
The Sopranos - 1x01 - Pilot.avi
The Sopranos - 1x02 - 46 Long.avi
The Sopranos - 1x03 - Denial, Anger, Acceptance.avi
The Sopranos - 1x04 - Meadowlands.avi
The Sopranos - 1x05 - College.avi
The Sopranos - 1x06 - Pax Soprana.avi
The Sopranos - 1x07 - Down Neck.avi
The Sopranos - 1x08 - The Legend of Tennessee Moltisanti.avi
The Sopranos - 1x09 - Boca.avi
The Sopranos - 1x10 - A Hit Is a Hit.avi
The Sopranos - 1x11 - Nobody Knows Anything.avi
The Sopranos - 1x12 - Isabella.avi
The Sopranos - 1x13 - I Dream of Jeannie Cusamano.avi
The Sopranos - 2x01 - Guy Walks into a Psychiatrist's Office.avi
The Sopranos - 2x02 - Do Not Resuscitate.avi
The Sopranos - 2x03 - Toodle-****ing-oo.avi
The Sopranos - 2x04 - Commendatori.avi
The Sopranos - 2x05 - Big Girls Don't Cry.avi
The Sopranos - 2x06 - The Happy Wanderer.avi
The Sopranos - 2x07 - D-Girl.avi
The Sopranos - 2x08 - Full Leather Jacket.avi
The Sopranos - 2x09 - From Where to Eternity.avi
The Sopranos - 2x10 - Bust-Out.avi
The Sopranos - 2x11 - House Arrest.avi
The Sopranos - 2x12 - The Knight in White Satin Armor.avi
The Sopranos - 2x13 - Funhouse.avi
The Sopranos - 3x01 - Mr. Ruggerio's Neighborhood.avi
The Sopranos - 3x02 - Proshai, Livushka.avi
The Sopranos - 3x03 - Fortunate Son.avi
The Sopranos - 3x04 - Employee of the Month.avi
The Sopranos - 3x05 - Another Toothpick.avi
The Sopranos - 3x06 - University.avi
The Sopranos - 3x07 - He Has Risen.avi
The Sopranos - 3x08 - Second Opinion.avi
The Sopranos - 3x09 - The Telltale Moozadell.avi
The Sopranos - 3x10 - To Save Us All From Satans Power.avi
The Sopranos - 3x11 - Pine Barrens.avi
The Sopranos - 3x12 - Amour Fou.avi
The Sopranos - 3x13 - Army of One.avi
The Sopranos - 4x01 - For All Debts Public and Private.avi
The Sopranos - 4x02 - No Show.avi
The Sopranos - 4x03 - Christopher.avi
The Sopranos - 4x04 - The Weight.avi
The Sopranos - 4x05 - Pie-o-My.avi
The Sopranos - 4x06 - Everybody Hurts.avi
The Sopranos - 4x07 - Watching Too Much Television.avi
The Sopranos - 4x08 - Mergers and Acquisitions.avi
The Sopranos - 4x09 - Whoever Did This.avi
The Sopranos - 4x10 - The Strong, Silent Type.avi
The Sopranos - 4x11 - Calling All Cars.avi
The Sopranos - 4x12 - Eloise.avi
The Sopranos - 4x13 - Whitecaps.avi
The Sopranos - 5x01 - Two Tonys.avi
The Sopranos - 5x02 - Rat Pack.avi
The Sopranos - 5x03 - Where's Johnny_.avi
The Sopranos - 5x04 - All Happy Families.avi
The Sopranos - 5x05 - Irregular Around The Margins.avi
The Sopranos - 5x06 - Sentimental Education.avi
The Sopranos - 5x07 - In Camelot.avi
The Sopranos - 5x08 - Marco Polo.avi
The Sopranos - 5x09 - Unidentified Black Male.avi
The Sopranos - 5x10 - Cold Cuts.avi
The Sopranos - 5x11 - The Test Dream.avi
The Sopranos - 5x12 - Long Term Parking.avi
The Sopranos - 5x13 - All Due Respect.avi
The Sopranos - 6x01 - Members Only.avi
The Sopranos - 6x02 - Join The Club.avi
The Sopranos - 6x03 - Mayham.avi
The Sopranos - 6x04 - The Fleshy Part of the Thigh.avi
The Sopranos - 6x05 - Mr. & Mrs. John Sacrimoni Request.avi
The Sopranos - 6x06 - Live Free or Die.avi
The Sopranos - 6x07 - Luxury Lounge.avi
The Sopranos - 6x08 - Johnny Cakes.avi
The Sopranos - 6x09 - The Ride.avi
The Sopranos - 6x10 - Moe n' Joe.avi
The Sopranos - 6x11 - Cold Stones.avi
The Sopranos - 6x12 - Kaisha.avi
The Sopranos - 6x13 -  Soprano Home Movies.avi
The Sopranos - 6x14 -  Stage 5.avi
The Sopranos - 6x15 - Remember When.avi
The Sopranos - 6x16 - Chasing It  .avi
The Sopranos - 6x17 - Walk Like a Man.avi
The Sopranos - 6x18 - Kennedy and Heidi.avi
The Sopranos - 6x19 - The Second Coming  .avi
The Sopranos - 6x20 - The Blue Comet.avi
The Sopranos - 6x21 - Made in America.avi
root@wonslung-raid:/mnt/raid/nas/Video/Television/Sopranos#

and it's rebuilding

Code:

root@wonslung-raid:/# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md0 : active raid5 sda1[0] sdf1[6] sdc1[5] sde1[3] sdd1[2] sdb1[1]
      4883799680 blocks level 5, 64k chunk, algorithm 2 [6/5] [UUUU_U]
      [>....................]  recovery =  0.7% (6929408/976759936) finish=311.8min speed=51836K/sec
      
unused devices: <none>

but when it hits 100% it fails again
also, heres as much other info as i can think to list

Code:

root@wonslung-raid:/# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58b8c - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/# mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58b9e - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/# mdadm --examine /dev/sdc1
/dev/sdc1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58bb6 - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     5       8       33        5      active sync   /dev/sdc1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/# mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58bc0 - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       49        2      active sync   /dev/sdd1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/# mdadm --examine /dev/sde1
/dev/sde1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58bd2 - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       65        3      active sync   /dev/sde1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/# mdadm --examine /dev/sdf1
/dev/sdf1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean
 Active Devices : 5
Working Devices : 6
 Failed Devices : 1
  Spare Devices : 1
       Checksum : bdd58be2 - correct
         Events : 1696722

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     6       8       81        6      spare   /dev/sdf1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
   2     2       8       49        2      active sync   /dev/sdd1
   3     3       8       65        3      active sync   /dev/sde1
   4     4       0        0        4      faulty removed
   5     5       8       33        5      active sync   /dev/sdc1
   6     6       8       81        6      spare   /dev/sdf1
root@wonslung-raid:/#

Code:

root@wonslung-raid:/# mdadm -D /dev/md0
/dev/md0:
        Version : 00.90
  Creation Time : Wed Jan  7 19:01:36 2009
     Raid Level : raid5
     Array Size : 4883799680 (4657.55 GiB 5001.01 GB)
  Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Fri Apr  3 06:50:51 2009
          State : clean, degraded, recovering
 Active Devices : 5
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

 Rebuild Status : 1% complete

           UUID : cec547d8:bb56daf4:cf7cba6b:ed693cc5
         Events : 0.1696722

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       6       8       81        4      spare rebuilding   /dev/sdf1
       5       8       33        5      active sync   /dev/sdc1
root@wonslung-raid:/#

and heres some of the kernel.log

Code:

Apr  3 01:07:53 wonslung-raid kernel: [10566.567158] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:07:53 wonslung-raid kernel: [10566.567240] ata4.00: BMDMA stat 0x64
Apr  3 01:07:53 wonslung-raid kernel: [10566.567285] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:07:53 wonslung-raid kernel: [10566.567286]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:07:53 wonslung-raid kernel: [10566.567449] ata4.00: status: { DRDY ERR }
Apr  3 01:07:53 wonslung-raid kernel: [10566.567492] ata4.00: error: { UNC }
Apr  3 01:07:54 wonslung-raid kernel: [10567.580734] ata4.00: configured for UDMA/133
Apr  3 01:07:54 wonslung-raid kernel: [10567.601696] ata4.01: configured for UDMA/133
Apr  3 01:07:54 wonslung-raid kernel: [10567.601808] ata4: EH complete
Apr  3 01:07:55 wonslung-raid kernel: [10568.574116] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:07:55 wonslung-raid kernel: [10568.574171] ata4.00: BMDMA stat 0x64
Apr  3 01:07:55 wonslung-raid kernel: [10568.574217] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:07:55 wonslung-raid kernel: [10568.574218]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:07:55 wonslung-raid kernel: [10568.574380] ata4.00: status: { DRDY ERR }
Apr  3 01:07:55 wonslung-raid kernel: [10568.574424] ata4.00: error: { UNC }
Apr  3 01:07:56 wonslung-raid kernel: [10569.581578] ata4.00: configured for UDMA/133
Apr  3 01:07:56 wonslung-raid kernel: [10569.600327] ata4.01: configured for UDMA/133
Apr  3 01:07:56 wonslung-raid kernel: [10569.600441] ata4: EH complete
Apr  3 01:07:57 wonslung-raid kernel: [10570.564521] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:07:57 wonslung-raid kernel: [10570.564577] ata4.00: BMDMA stat 0x64
Apr  3 01:07:57 wonslung-raid kernel: [10570.564622] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:07:57 wonslung-raid kernel: [10570.564623]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:07:57 wonslung-raid kernel: [10570.564785] ata4.00: status: { DRDY ERR }
Apr  3 01:07:57 wonslung-raid kernel: [10570.564829] ata4.00: error: { UNC }
Apr  3 01:07:58 wonslung-raid kernel: [10571.560325] ata4.00: configured for UDMA/133
Apr  3 01:07:58 wonslung-raid kernel: [10571.581578] ata4.01: configured for UDMA/133
Apr  3 01:07:58 wonslung-raid kernel: [10571.581692] ata4: EH complete
Apr  3 01:07:59 wonslung-raid kernel: [10572.571512] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:07:59 wonslung-raid kernel: [10572.571568] ata4.00: BMDMA stat 0x64
Apr  3 01:07:59 wonslung-raid kernel: [10572.571613] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:07:59 wonslung-raid kernel: [10572.571614]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:07:59 wonslung-raid kernel: [10572.571776] ata4.00: status: { DRDY ERR }
Apr  3 01:07:59 wonslung-raid kernel: [10572.571820] ata4.00: error: { UNC }
Apr  3 01:08:00 wonslung-raid kernel: [10573.570323] ata4.00: configured for UDMA/133
Apr  3 01:08:00 wonslung-raid kernel: [10573.591577] ata4.01: configured for UDMA/133
Apr  3 01:08:00 wonslung-raid kernel: [10573.591684] ata4: EH complete
Apr  3 01:08:01 wonslung-raid kernel: [10574.537036] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:08:01 wonslung-raid kernel: [10574.537092] ata4.00: BMDMA stat 0x64
Apr  3 01:08:01 wonslung-raid kernel: [10574.537137] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:08:01 wonslung-raid kernel: [10574.537138]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:08:01 wonslung-raid kernel: [10574.537300] ata4.00: status: { DRDY ERR }
Apr  3 01:08:01 wonslung-raid kernel: [10574.537344] ata4.00: error: { UNC }
Apr  3 01:08:02 wonslung-raid kernel: [10575.540326] ata4.00: configured for UDMA/133
Apr  3 01:08:02 wonslung-raid kernel: [10575.561577] ata4.01: configured for UDMA/133
Apr  3 01:08:02 wonslung-raid kernel: [10575.561685] ata4: EH complete
Apr  3 01:08:03 wonslung-raid kernel: [10576.519154] ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
Apr  3 01:08:03 wonslung-raid kernel: [10576.519209] ata4.00: BMDMA stat 0x64
Apr  3 01:08:03 wonslung-raid kernel: [10576.519255] ata4.00: cmd 25/00:c0:3f:7f:c7/00:01:45:00:00/e0 tag 0 dma 229376 in
Apr  3 01:08:03 wonslung-raid kernel: [10576.519256]          res 51/40:00:39:80:c7/40:00:45:00:00/e0 Emask 0x9 (media error)
Apr  3 01:08:03 wonslung-raid kernel: [10576.519418] ata4.00: status: { DRDY ERR }
Apr  3 01:08:03 wonslung-raid kernel: [10576.519462] ata4.00: error: { UNC }
Apr  3 01:08:04 wonslung-raid kernel: [10577.510324] ata4.00: configured for UDMA/133
Apr  3 01:08:04 wonslung-raid kernel: [10577.531576] ata4.01: configured for UDMA/133
Apr  3 01:08:04 wonslung-raid kernel: [10577.531705] sd 3:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK
Apr  3 01:08:04 wonslung-raid kernel: [10577.531710] sd 3:0:0:0: [sdc] Sense Key : Medium Error [current] [descriptor]
Apr  3 01:08:04 wonslung-raid kernel: [10577.531714] Descriptor sense data with sense descriptors (in hex):
Apr  3 01:08:04 wonslung-raid kernel: [10577.531716]         72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
Apr  3 01:08:04 wonslung-raid kernel: [10577.531725]         45 c7 80 39
Apr  3 01:08:04 wonslung-raid kernel: [10577.531729] sd 3:0:0:0: [sdc] Add. Sense: Unrecovered read error - auto reallocate failed
Apr  3 01:08:04 wonslung-raid kernel: [10577.531734] end_request: I/O error, dev sdc, sector 1170702393
Apr  3 01:08:04 wonslung-raid kernel: [10577.531787] raid5:md0: read error not correctable (sector 1170702328 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531790] raid5: Disk failure on sdc1, disabling device.
Apr  3 01:08:04 wonslung-raid kernel: [10577.531791] raid5: Operation continuing on 4 devices.
Apr  3 01:08:04 wonslung-raid kernel: [10577.531891] raid5:md0: read error not correctable (sector 1170702336 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531894] raid5:md0: read error not correctable (sector 1170702344 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531897] raid5:md0: read error not correctable (sector 1170702352 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531900] raid5:md0: read error not correctable (sector 1170702360 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531903] raid5:md0: read error not correctable (sector 1170702368 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531905] raid5:md0: read error not correctable (sector 1170702376 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531908] raid5:md0: read error not correctable (sector 1170702384 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531911] raid5:md0: read error not correctable (sector 1170702392 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531914] raid5:md0: read error not correctable (sector 1170702400 on sdc1).
Apr  3 01:08:04 wonslung-raid kernel: [10577.531934] ata4: EH complete
Apr  3 01:08:04 wonslung-raid kernel: [10577.533758] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB)
Apr  3 01:08:04 wonslung-raid kernel: [10577.536166] sd 3:0:0:0: [sdc] Write Protect is off
Apr  3 01:08:04 wonslung-raid kernel: [10577.536168] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Apr  3 01:08:04 wonslung-raid kernel: [10577.545334] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr  3 01:08:04 wonslung-raid kernel: [10577.650139] md: md0: recovery done.
Apr  3 01:08:04 wonslung-raid kernel: [10577.654418] sd 3:0:1:0: [sdd] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB)
Apr  3 01:08:04 wonslung-raid kernel: [10577.655281] sd 3:0:1:0: [sdd] Write Protect is off
Apr  3 01:08:04 wonslung-raid kernel: [10577.655284] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
Apr  3 01:08:04 wonslung-raid kernel: [10577.655314] sd 3:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr  3 01:08:04 wonslung-raid kernel: [10577.655345] sd 3:0:0:0: [sdc] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB)
Apr  3 01:08:04 wonslung-raid kernel: [10577.655360] sd 3:0:0:0: [sdc] Write Protect is off
Apr  3 01:08:04 wonslung-raid kernel: [10577.655362] sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
Apr  3 01:08:04 wonslung-raid kernel: [10577.655388] sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr  3 01:08:04 wonslung-raid kernel: [10577.655415] sd 3:0:1:0: [sdd] 1953525168 512-byte hardware sectors: (1.00 TB/931 GiB)
Apr  3 01:08:04 wonslung-raid kernel: [10577.655429] sd 3:0:1:0: [sdd] Write Protect is off
Apr  3 01:08:04 wonslung-raid kernel: [10577.655432] sd 3:0:1:0: [sdd] Mode Sense: 00 3a 00 00
Apr  3 01:08:04 wonslung-raid kernel: [10577.655457] sd 3:0:1:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Apr  3 01:08:04 wonslung-raid kernel: [10577.679398] RAID5 conf printout:
Apr  3 01:08:04 wonslung-raid kernel: [10577.679400]  --- rd:6 wd:4
Apr  3 01:08:04 wonslung-raid kernel: [10577.679402]  disk 0, o:1, dev:sda1
Apr  3 01:08:04 wonslung-raid kernel: [10577.679404]  disk 1, o:1, dev:sdb1
Apr  3 01:08:04 wonslung-raid kernel: [10577.679406]  disk 2, o:1, dev:sdd1
Apr  3 01:08:04 wonslung-raid kernel: [10577.679407]  disk 3, o:1, dev:sde1
Apr  3 01:08:04 wonslung-raid kernel: [10577.679409]  disk 4, o:1, dev:sdf1
Apr  3 01:08:04 wonslung-raid kernel: [10577.679410]  disk 5, o:0, dev:sdc1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711256] RAID5 conf printout:
Apr  3 01:08:04 wonslung-raid kernel: [10577.711259]  --- rd:6 wd:4
Apr  3 01:08:04 wonslung-raid kernel: [10577.711261]  disk 0, o:1, dev:sda1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711263]  disk 1, o:1, dev:sdb1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711264]  disk 2, o:1, dev:sdd1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711266]  disk 3, o:1, dev:sde1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711268]  disk 5, o:0, dev:sdc1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711275] RAID5 conf printout:
Apr  3 01:08:04 wonslung-raid kernel: [10577.711276]  --- rd:6 wd:4
Apr  3 01:08:04 wonslung-raid kernel: [10577.711278]  disk 0, o:1, dev:sda1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711279]  disk 1, o:1, dev:sdb1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711281]  disk 2, o:1, dev:sdd1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711283]  disk 3, o:1, dev:sde1
Apr  3 01:08:04 wonslung-raid kernel: [10577.711284]  disk 5, o:0, dev:sdc1
Apr  3 01:08:04 wonslung-raid kernel: [10577.741256] RAID5 conf printout:
Apr  3 01:08:04 wonslung-raid kernel: [10577.741258]  --- rd:6 wd:4
Apr  3 01:08:04 wonslung-raid kernel: [10577.741260]  disk 0, o:1, dev:sda1
Apr  3 01:08:04 wonslung-raid kernel: [10577.741262]  disk 1, o:1, dev:sdb1
Apr  3 01:08:04 wonslung-raid kernel: [10577.741264]  disk 2, o:1, dev:sdd1
Apr  3 01:08:04 wonslung-raid kernel: [10577.741265]  disk 3, o:1, dev:sde1

i'm still kind of a newb...i know enough to do everyday stuff...and more than your total newb...but i still don't know linux kong fu like some of you

i've searched google all night and tried everything i know to try short of buying some new drives and moving data

**wonslung** · 04-04-2009

well i wasn't able to get any help across about 5 forums.....all i've just moved as much data as i can to other drives and i guess i'm going to be forced to zero the superblocks and start over
really noone knows why it would let me force start it but fail as soon as it hit 100% on a resync?
it let me copy every bit of data to other drives or computers via the network so long as i force start it degraded....but it won't rebuild with any drive i throw at it..
infact i ended up using the drive that wouldn't resync to move 900GB's off the array

**adsllc** · 04-16-2009

Disclaimer:
The following could be completely wrong and cause you to lose all your data - proceed with backups and at your own risk. I'm not going to take responsibility of ~1TB of data!

Short version:
Your sdc is bad - replace it. Partition the new drive then run "mdadm /dev/md0 -a /dev/sdc1".

Long version:
You can tell that sdc is bad several ways:

1. The kernel.org messages like "raid5:md0: read error not correctable (sector 1170702336 on sdc1)" and
2. "cat /proc/mdstat" has the fifth U marked as an underline and that drive is listed as sdc1[5], or
3. Learn to love smartctl.

Use "hdparm -i /dev/sdc" to get the serial number of the drive so that you are sure to replace the correct one.

Next - you are using the wrong mdadm mode. "Assembling" the array tells the system which device nodes (ie /dev/sda1) make up /dev/md0 and brings it online.

You aren't changing which device nodes make up /dev/md0, you are changing the hardware behind the device node. In other words, the new drive will still be sdc1 and /dev/md0 will still be made of /dev/sd[a-f]1. By forcing the system to assemble you are simply "clearing FAULTY flag for ... /dev/sdc1" as a side-effect, which is then letting the array attempt to rebuild. Since sdc IS actually faulty, the rebuild fails.

The correct thing is to replace sdc, create the partitions to match the other drives (don't forget to use partition type=fd [software raid]), then use the following "Manage" command to hot add the new drive into the array:

mdadm /dev/md0 -a /dev/sdc1

You may ask wonder why this drive happened to fail exactly during a power failure? Why didn't the RAID see the failure before? The answer is that RAID doesn't know the sector is bad until it tries to read it - which it had no reason to do until it rebooted. Lurking bad sectors like this are a problem. I have seen mirrors where both drives had lurking bad sectors and the RAID was perfectly happy until suddenly it was completely broken. RAID1 is somewhat forgiving about this since you can mount each drive individually - it's practically a data death sentence for RAID5.

You can detect these lurking sectors by using something like "badblocks /dev/sd[a-f]" from time to time, using "smartctl -t long /dev/sd[a-f]" followed by "smartctl -t selftest /dev/sd[a-f]" after it completes occasionally, or ideally by automating the previous using smartd.

**slyt** · 04-22-2009

I am very sorry but due to new posts limitation I put my case here.

At the beginning ....

[root@smb ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 hde1[0] hdf1[1]
244195904 blocks [2/2] [UU]
[==>..................] resync = 13.8% (33701824/244195904) finish=613378.6min speed=5K/sec

unused devices: <none>
[root@smb ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid1 hde1[0] hdf1[1]
244195904 blocks [2/2] [UU]
[==>..................] resync = 13.8% (33701824/244195904) finish=613646.4min speed=5K/sec

unused devices: <none>

then ...

[root@smb ~]# mdadm --misc -D /dev/md0
/dev/md0:
Version : 00.90.03
Creation Time : Wed Sep 29 19:21:49 2004
Raid Level : raid1
Array Size : 244195904 (232.88 GiB 250.06 GB)
Device Size : 244195904 (232.88 GiB 250.06 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Persistence : Superblock is persistent

Update Time : Wed Apr 22 16:50:00 2009
State : dirty, recovering
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

Rebuild Status : 13% complete

Number Major Minor RaidDevice State
0 33 1 0 active sync /dev/hde1
1 33 65 1 active sync /dev/hdf1
UUID : c9a5bc46:71666021:9085eb7b:e49d0e9d
Events : 0.119

The problem is that rebuild process freezes on 13,8% and I have no access to /home which is my /dev/md0 .

In my opinion one of the disks failed but which one and what next ...

Any help is appreciated ...

Regards

**adsllc** · 04-22-2009

Do

Code:

smartctl -t long /dev/hde
smartctl -t long /dev/hdf

Wait a few hours for it to complete, then

Code:

smartctl -l selftest /dev/hde
smartctl -l selftest /dev/hdf

to see which drive failed. You can also try

Code:

grep "I/O error" /var/log/messages*

or

Code:

dd if=/dev/hde of=/dev/null bs=1024 skip=33701500 count=1000
dd if=/dev/hdf of=/dev/null bs=1024 skip=33701500 count=1000

to see which one fails.

Replace the drive and go from there.

You should be able to mount the array even with a failed drive; what happens when you try "mount /dev/md0 /home"?

**slyt** · 04-25-2009

Thank you for your fast and accurate answer.

I did

Code:

smartctl -t quick /dev/hde
smartctl -t quick /dev/hdf

then

Code:

smartctl -l selftest /dev/hde
smartctl -l selftest /dev/hdf

and everything appeared to be clear. My /dev/hde drive failed (8-months old WDC SE 250GB).

**fentonc** · 04-29-2009

I have the similar problem after power failure. My configuration is 3x1TB Raid5, through SATA-USB converter.
After power failure, I restarted the PC, remount RAID devices / ext3 and performed f2sck. The ext3 journal was recovered and seems no problem. mdadm was resync 3 discs in background.
After few hours, the RAID disks became un-readable and dropped in the "Sense Key: no sense" infinity loop. The sync stopped at 23.8%. After kernel upgrade & misc patch deployed, the problem was still unable to be solved.
Performs badblocks reading test, 2 discs hanged at 23.8%.

Unplug those problem discs and attached to XP, use "HDD Generator" to scan and repair logical bad sectors. Total 784 bad sectors were found and repaired. Remount those disc through mdadm, the resync completed successfully.

Thread Tools

Display

Mdadm problems.....i'm dying please help

Fixing your array - replace /dev/sdc

Linux RAID problem.

Posting Permissions