F-16 mdadm create 4 3tb RAID 5 array

[v8esprit]

(Before I begin, let me apologize for the use of the open parenthesis. Turns out the forum software thought I was embedding URLS in my post! Pretty silly on a Linux site.)

All,
I've been struggling for quite a while with this but I'm very close to solving this problem.

I am using a Zotac H67-ITX mobo with Fedora 16. I have upgraded the Intel OROM RST to 10.8 so I have RAID support of drives larger than 3TB. Hardware isn't an issue. (if it is, let's first try to resolve all software issues first.

So, RAID 5, 4 drives, I'll have access to the disk capacity of all minus 1. So quick basic math:

3TB * 4 = 12TB - 3TB = 9TB. I'm expecting an array of 9tb when this is all done with.

I started by blowing out the superblocks on all of these drives:

[root}lox-itx ~]# dd if={dev{zero of={dev{sdb bs=1 count=2048
2048+0 records in
2048+0 records out
2048 bytes (2.0 kB) copied, 0.00702857 s, 291 kB{s
[root}lox-itx ~]# dd if={dev{zero of={dev{sdc bs=1 count=2048
2048+0 records in
2048+0 records out
2048 bytes (2.0 kB) copied, 0.00266559 s, 768 kB{s
[root}lox-itx ~]# dd if={dev{zero of={dev{sdd bs=1 count=2048
2048+0 records in
2048+0 records out
2048 bytes (2.0 kB) copied, 0.00435692 s, 470 kB{s
[root}lox-itx ~]# dd if={dev{zero of={dev{sde bs=1 count=2048
2048+0 records in
2048+0 records out
2048 bytes (2.0 kB) copied, 0.00261023 s, 785 kB{s
[root}lox-itx ~]#

Let's get this party started:

[root}lox-itx ~]# mdadm --create {dev{md0 --verbose --chunk=128 --level=raid5 --raid-devices=4 {dev{sdb {dev{sdc {dev{sdd {dev{sde
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: size set to 2930265344K
mdadm: Defaulting to version 1.2 metadata
mdadm: array {dev{md0 started.
[root}lox-itx ~]# cat {proc{mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sde[4] sdd[2] sdc[1] sdb[0]
8790796032 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4{3] [UUU_]
[>....................] recovery = 0.1% (3305656{2930265344) finish=797.8min speed=61141K{sec

unused devices: <none>

Ok, the array was made and it is busy syncing (initializing) it.

Bad news, it set it as a 3TB size... now that is not what I want or am expecting. I query the array status:

[root}lox-itx ~]# mdadm --detail {dev{md0
{dev{md0:
Version : 1.2
Creation Time : Tue Dec 6 14:07:03 2011
Raid Level : raid5
Array Size : 8790796032 (8383.56 GiB 9001.78 GB)
Used Dev Size : 2930265344 (2794.52 GiB 3000.59 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Tue Dec 6 14:07:04 2011
State : clean, degraded, recovering
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 128K

Rebuild Status : 0% complete

Name : lox-itx.action-unltd.com:0 (local to host lox-itx.action-unltd.com)
UUID : 9ca54dcd:c87c676b:26b4853a:fa907eb1
Events : 1

Number Major Minor RaidDevice State
0 8 16 0 active sync {dev{sdb
1 8 32 1 active sync {dev{sdc
2 8 48 2 active sync {dev{sdd
4 8 64 3 spare rebuilding {dev{sde

-----------------
Now wait a minute... 2 things have me confused. Maybe you can help me understand this. Take a close look at the following lines.

Array Size : 8790796032 (8383.56 GiB 9001.78 GB)
Used Dev Size : 2930265344 (2794.52 GiB 3000.59 GB)

It states the array size it 9tb (what I want) and what does Used Dev Size mean?

Also you see the last line? It claims that disk4 is acting as a spare!? I always thought a spare, or hot spare, was there, laying dormant, waiting to be brought alive when an array member disk failed and the array would be rebuilt on it, while the bad{failed disk was replaced (to take the place of the new spare) I actually don't want a (hot)spare. You guy know how much a 3TB hard disk costs these days? (Thai Flood)

Well, then I say to myself, let's check the usable space:

[root}lox-itx ~]# parted {dev{md0
GNU Parted 3.0
Using {dev{md0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p
Error: {dev{md0: recognized disk label
Model: Linux Software RAID Array (md)
Disk {dev{md0: 9002GB
Sector size (logical{physical): 512B{4096B
Partition Table: unknown
(parted)

Holy cow, I got my 9TB! I guess I should be happy and I'm not at my wits end... (Told you I was close) Any of you guys familiar with mdadm (& large disks) and why it is reporting confusing info to me?

Thanks,
Lox

[drl]

Hi.

This suggests an answer:

       When creating a RAID5 array, mdadm will automatically create a degraded
       array with an extra spare drive.  This is because building the spare
       into a degraded array is in general faster than resyncing the parity on
       a non-degraded, but not clean, array.  This feature can be overridden
       with the --force option.

-- excerpt from man mdadm

cheers, drl