Hi Forums

I have a software raid 5 on my server with 8x 320GB. It has worked fine for 4 months, now it failed somehow on 2 power failures. (Yes I have a UPS but the battery seems to be very weak).

When I try to assemble it, I get the following:
Code:
[root@wrdlbrnft2 ~]# mdadm --assemble --force --verbose /dev/md0 /dev/sd[b-i]
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdc is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdd is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sde is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdf is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdg is identified as a member of /dev/md0, slot 9.
mdadm: /dev/sdh is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sdi is identified as a member of /dev/md0, slot 7.
mdadm: added /dev/sdc to /dev/md0 as 1
mdadm: added /dev/sdd to /dev/md0 as 2
mdadm: added /dev/sde to /dev/md0 as 3
mdadm: added /dev/sdf to /dev/md0 as 4
mdadm: no uptodate device for slot 5 of /dev/md0
mdadm: no uptodate device for slot 6 of /dev/md0
mdadm: added /dev/sdi to /dev/md0 as 7
mdadm: added /dev/sdh to /dev/md0 as 8
mdadm: added /dev/sdg to /dev/md0 as 9
mdadm: added /dev/sdb to /dev/md0 as 0
mdadm: /dev/md0 assembled from 6 drives and 2 spares - not enough to start the array.
Why are the disks marked as spare disk? With mdadm --examine they show up the correct major and minor number. So I assumed that on 2 disks the Event Counter were lower than on the other drives, but they aren't:
Code:
[root@wrdlbrnft2 ~]# mdadm --examine /dev/md0 /dev/sd[b-i]
/dev/sdb:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 394006f5 - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8       16        0      active sync   /dev/sdb

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdc:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 39400707 - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       32        1      active sync   /dev/sdc

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdd:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 39400719 - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     2       8       48        2      active sync   /dev/sdd

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sde:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 3940072b - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     3       8       64        3      active sync   /dev/sde

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdf:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 3940073d - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     4       8       80        4      active sync   /dev/sdf

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 39400751 - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     9       8       96        9      spare   /dev/sdg

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdh:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 3940075f - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     8       8      112        8      spare   /dev/sdh

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
/dev/sdi:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : 707991be:b7f10572:3177a144:9624f8a6
  Creation Time : Fri May 11 17:49:35 2007
     Raid Level : raid5
  Used Dev Size : 312571136 (298.09 GiB 320.07 GB)
     Array Size : 2187997952 (2086.64 GiB 2240.51 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Tue Sep 18 09:35:43 2007
          State : clean
 Active Devices : 6
Working Devices : 8
 Failed Devices : 2
  Spare Devices : 2
       Checksum : 39400773 - correct
         Events : 0.1801958

         Layout : left-symmetric
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     7       8      128        7      active sync   /dev/sdi

   0     0       8       16        0      active sync   /dev/sdb
   1     1       8       32        1      active sync   /dev/sdc
   2     2       8       48        2      active sync   /dev/sdd
   3     3       8       64        3      active sync   /dev/sde
   4     4       8       80        4      active sync   /dev/sdf
   5     5       0        0        5      faulty removed
   6     6       0        0        6      faulty removed
   7     7       8      128        7      active sync   /dev/sdi
   8     8       8      112        8      spare   /dev/sdh
   9     9       8       96        9      spare   /dev/sdg
So what could I do to get my array back working? Maybe I'll have some data loss but if I get back 90% I'd be happy already.
I have no Idea what could be wrong with my array, cause every single drive has the same event counter and show status as 'clean'

plz help!