mkfs.ext3 failing/failed disk errors and mdadm

[javaholic]

I am having trouble with my Debian box that used to be my file server.

the story goes as follows:

I once had a three disk mdadm created raid5 array that failed one day so i bought three more (not as direct replacements and all from ebay) to help solve the failure more than anything. From what i could gather at the time two drives had failed. I wasn't too sure but couldn't diagnose the fault. I had been considering adding to my array with a fourth drive. So i took out what i thought was the two faulty drives i thought were at fault. I still queston this as they are not much more than 3-4 months old and i still have drives that work that are in my possession that are about 10-13 years old but obviously not much good for storage anymore of course. All this time i had been using mdadm to create my array.

Having removed what i thought were the faulty drives and zeroed them all all was going well till after about a day went past of the create process when i noticed using cat/ /proc/mdstat reported one failed drive reported and one spare drive. running the create option with --assume-clean option meant that mdadm successfully managed to create the array however when i try and run mkfs.ext3 /dev/md0 to the newly created storage space in question i get the following:

mke2fs 1.40-WIP (14-Nov-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
366297088 inodes, 732569952 blocks
36628497 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
22357 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544

Writing inode tables: ^[[A^[[Adone
ext2fs_mkdir: Attempt to read block from filesystem resulted in short read while creating root dir

please note the ^[[A is registered keystrokes as i have been monitoring it's progress which takes some amount of time as you can probably imagine.

After entering cat /proc/mdstat into console i get a report that states that i have one spare drive and one failed drive which after reboot looks like this with named drives removed as just the two remaining drives intact as you and I might expect.

In case it helps i have included the contents of my dmesg which reference all drives in a hope to help diagnose problem quicker.

sata_sil 0000:00:0b.0: Applying R_ERR on DMA activate FIS errata fix
8139cp: 10/100 PCI Ethernet driver v1.2 (Mar 22, 2004)
ata1: SATA max UDMA/100 cmd 0xF8820080 ctl 0xF882008A bmdma 0xF8820000 irq 10
ata2: SATA max UDMA/100 cmd 0xF88200C0 ctl 0xF88200CA bmdma 0xF8820008 irq 10
ata3: SATA max UDMA/100 cmd 0xF8820280 ctl 0xF882028A bmdma 0xF8820200 irq 10
ata4: SATA max UDMA/100 cmd 0xF88202C0 ctl 0xF88202CA bmdma 0xF8820208 irq 10
scsi0 : sata_sil
usb 1-1: new full speed USB device using ohci_hcd and address 2
usb 1-1: configuration #1 chosen from 1 choice
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata1.00: ata1: dev 0 multi count 16
ata1.00: configured for UDMA/100
scsi1 : sata_sil
usb 1-2: new low speed USB device using ohci_hcd and address 3
usb 1-2: configuration #1 chosen from 1 choice
usbcore: registered new driver hiddev
input: USBPS2 as /class/input/input0
input: USB HID v1.00 Keyboard [USBPS2] on usb-0000:00:02.0-2
input: USBPS2 as /class/input/input1
input: USB HID v1.00 Mouse [USBPS2] on usb-0000:00:02.0-2
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata2.00: ata2: dev 0 multi count 16
ata2.00: configured for UDMA/100
scsi2 : sata_sil
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata3.00: ATA-7, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata3.00: ata3: dev 0 multi count 16
ata3.00: configured for UDMA/100
scsi3 : sata_sil
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata4.00: ATA-8, max UDMA/133, 1953525168 sectors: LBA48 NCQ (depth 0/32)
ata4.00: ata4: dev 0 multi count 16
ata4.00: configured for UDMA/100
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-32Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
sda: Write Protect is off
sda: Mode Sense: 00 3a 00 00
SCSI device sda: drive cache: write back
sda: sda1
sd 0:0:0:0: Attached scsi disk sda
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
sdb: Write Protect is off
sdb: Mode Sense: 00 3a 00 00
SCSI device sdb: drive cache: write back
sdb: sdb1
sd 1:0:0:0: Attached scsi disk sdb
SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
SCSI device sdc: 1953525168 512-byte hdwr sectors (1000205 MB)
sdc: Write Protect is off
sdc: Mode Sense: 00 3a 00 00
SCSI device sdc: drive cache: write back
sdc: sdc1
sd 2:0:0:0: Attached scsi disk sdc
SCSI device sdd: 1953525168 512-byte hdwr sectors (1000205 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: drive cache: write back
SCSI device sdd: 1953525168 512-byte hdwr sectors (1000205 MB)
sdd: Write Protect is off
sdd: Mode Sense: 00 3a 00 00
SCSI device sdd: drive cache: write back
sdd: sdd1
sd 3:0:0:0: Attached scsi disk sdd
raid5: automatically using best checksumming function: pIII_sse
pIII_sse : 4010.000 MB/sec
raid5: using function: pIII_sse (4010.000 MB/sec)
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
raid6: int32x1 583 MB/s
raid6: int32x2 708 MB/s
raid6: int32x4 554 MB/s
raid6: int32x8 401 MB/s
raid6: mmxx1 1216 MB/s
raid6: mmxx2 1931 MB/s
raid6: sse1x1 1120 MB/s
raid6: sse1x2 1910 MB/s
raid6: using algorithm sse1x2 (1910 MB/s)
md: raid6 personality registered for level 6
md: raid5 personality registered for level 5
md: raid4 personality registered for level 4
md: md0 stopped.
md: bind<sda1>
md: bind<sdc1>
md: bind<sdd1>
md: bind<sdb1>
md: kicking non-fresh sdc1 from array!
md: unbind<sdc1>
md: export_rdev(sdc1)
md: kicking non-fresh sda1 from array!
md: unbind<sda1>
md: export_rdev(sda1)
raid5: device sdb1 operational as raid disk 1
raid5: device sdd1 operational as raid disk 3
raid5: not enough operational devices for md0 (2/4 failed)
RAID5 conf printout:
--- rd:4 wd:2 fd:2
disk 1, o:1, dev:sdb1
disk 3, o:1, dev:sdd1
raid5: failed to run raid set md0
md: pers->run() failed ...

Sata card i am using is a Fake raid silicon Image 31** (i think 3114) with four sata ports which had three 1TB attached making a total storage of 1.7/8TB of data which worked fine. With the failure trying to create a bigger(2.8TB) i did try and revert back to creating only a three disk (1.7TB) array thinking that a lack of default support in the kernel for GPT might be causing the problem but this has also failed in a similar way. Other things i have tried was creating a deb file from a fedora 10 RPM using alien to install libparted 1.8.8 believing that the fault for my problems may be related to a reasonably well know bug in earlier (1.7.1) versions of parted.

The behaviour of MDADM is still a bit too alien for me and can't/don't understand why it is failing to work. the most annoying part of this for me is more financial as these replacement drives were and are still pretty expensive and to still not come up with a solution does not make me happy.

As a slight aside i would be very interested in a learning or reading a tutorial that goes into adding GPT support to the Debian if you know of one freely available on the net.

Despite my love for Debian for its stability if i thought that a better different and ultimately working storage solution could be got from the use of any distro other at this moment in time i would use it. Not getting this to work as it should is a deal breaker for future Debian use.

Post Scriptum: If you can think of a better heading for this discussion please don't hesitate to inform me about it.

[iwanabeguru]

post the outputs of
fdisk -l
cat /proc/mdstat
and the content of /etc/mdadm/mdadm.conf

[javaholic]

At what stage of the process?

And i should say i have never got to the stage where i have been happy with the config/build/create process so much so as to create the mdadm.conf file.

[iwanabeguru]

so you have a damaged raid5 array.
If it is built then the /proc/mdstat contains some information.
The information what you have given good for nothing.

[javaholic]

Bar when i had the three disk raid 5 array working with the original setup. It was working then.