[SOLVED] Fedora Software RAID 5 Setup

[rdyas]

Hi All

I'm new to linux admin, but thought I'd leap in the deep end and have just set up my first home server - HL Microserver N36L. Installed Vortexbox which is based on Fedora release 14 and put 4 x 1TB drives in.

Then created a raid on these disks using:

mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

more /proc/mdstat showed a 512k chunk size, so I formatted as follows:

mkfs –t xfs –d su=512k –d sw=3 /dev/md0

but format told me that maximum chunk size is 32k and was defaulting to that.

The Question: It is up and running - well mdstat has [UUUU] and I can use it OK, but is the above sensible? In particular, should I adjust any of the raid parameters to improve performance - not sure how you measure that?

thanks.

[Rubberman]

You really don't need to specify chuck size to mkfs except in rare situations. The "chunk size" reported by mdadm is the device sector size, not the file system parameter. In any case, the mkfs parameter (32K) is the minimum file/directory allocation size.

[rdyas]

OK thanks, so I dont have to worry too much about chunksize.

I have just done some tests copying large files to and from my server and I am getting approx. 75 MB/sec reading and 12MB/sec writing - is this what I should expect in terms of performance?

thanks
Roger.

[Rubberman]

OK thanks, so I dont have to worry too much about chunksize.

I have just done some tests copying large files to and from my server and I am getting approx. 75 MB/sec reading and 12MB/sec writing - is this what I should expect in terms of performance?

thanks
Roger.

Whether or not this is reasonable performance to expect depends a lot on the controller, speed of disc interfaces, system memory, drive characteristics, CPU speed, number of CPU cores, file system type, etc. In any case, for RAID-5, write operations will be slower than read operations because of the time required to calculate and write the parity blocks. Reading is much faster because of the striping used for RAID-5 so that parallel seek and read operations can be done spread over the discs.

Example:

I have an 8 core 3GHz system with an eSata (Sata2) array configured as RAID-5 running 4 Seagate 1.5TB Barracuda discs (7200rpm). Read speed peaks at about 220MBps (megabytes - about 1.7gbps), when copying a file from the array to a 7200rpm WD disc that is also on an eSata (Sata2) link. The copy stabilizes at somewhere between 100 and 125MBps, or 1gbps. This is a 40 gigabyte file (disc backup image). Both drives are using an ext4 file system. The file copy took less than 6 minutes real time (1minute 18seconds system time). Going the other way, it stabilized out at about 90MBps (720mbps) write speed. The actual write performance, including the 30% overhead for writing parity data (1 sector parity for 3 sectors data) was about 120MBps (almost 1gbps), which calculates reasonably well - 90MBps data + 30MBps parity == 120MBps total disc utilization. So, writing to the array has a 30% overhead attached to it. Given this, I would expect your array to get about 50MBps write speed for data, so we need to look at other factors, such as speed of the device you are writing from (how are you generating your data), the controller, disc characteristics, etc.

[rdyas]

Thanks for all the help Rubberman, much appreciated!

My setup is a Dell XPS 420 Intel Core2 Duo with 4Gig RAM running fresh install of Windows 7 and 1Gig internal Seagate drive connected by ethernet via Netgear Gigabit switch to HP N36L Microserver with 1Gig RAM, AMD Athlon 1.3 GHz, with 4 x 1T Seagate drives in a Raid 5 array, ethernet card in server is NetXtreme BCM5723 Gigabit Ethernet PCIe.

dmadm -D /dev/md127 gives output below.

Can you see a weak link, or am I doing as well as can be expected from a fairly low spec server?

thanks
Roger


Version : 1.2
Creation Time : Fri Apr 29 17:40:11 2011
Raid Level : raid5
Array Size : 2930284032 (2794.54 GiB 3000.61 GB)
Used Dev Size : 976761344 (931.51 GiB 1000.20 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon May 9 21:22:03 2011
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : vortexbox.localdomain:0 (local to host vortexbox.localdomain)
UUID : a0a5a6ce:891cdc2e:976c97f8:7104f751
Events : 1307

Number Major Minor RaidDevice State
0 8 0 0 active sync /dev/sda
1 8 16 1 active sync /dev/sdb
2 8 32 2 active sync /dev/sdc
4 8 48 3 active sync /dev/sdd

[Rubberman]

Ok. So basically your Windows system is connected to a NAS (Network Attached Storage) unit running Linux. I have a couple of those (Buffalo TeraStation units), and even with the gigabit ethernet connections to my workstation via a Cisco gigabit switch, I am lucky to get 100mbps (megabits, not megabytes per second) throughput. My examples were on RAID and drive units connected directly to my workstation/server. You are getting considerably better performance than that (75MBps == 600mbps vs. my TeraStation 100mbps), and since the gigabit connection maxes out at 1000mbps, if you consider network and task switching overhead, 600mbps is pretty decent for reads. The writes will depend a lot upon other factors that we haven't investigated yet. I'll have to think about how you might be able to increase your write performance and get back to you on that.

[meton_magis]

Example:

I have an 8 core 3GHz system with an eSata (Sata2) array configured as RAID-5 running 4 Seagate 1.5TB Barracuda discs (7200rpm).

I have a couple of those (Buffalo TeraStation units), and even with the gigabit ethernet connections to my workstation via a Cisco gigabit switch,

What, I don't believe you. You can't have all those cool toys, you're married.

[Rubberman]

What, I don't believe you. You can't have all those cool toys, you're married.

Har-de-har-har... That's because I'm an IT and systems engineering consultant. My wife is a physicist. She has 2 17" Apple Macbook Pros, two printers, several firewire attached external discs (including a couple of portable units), and a bunch of other tech toys. Let's see what we have:

Linux/Unix-based Systems:

1 Dell D630 dual-core Laptop running Ubuntu Linux (my travel machine)
2 OLPC laptops running Linux (a version of Ubuntu/Debian) and Sugar
1 Toshiba netbook running ... wait for it ... Fermi Linux
2 17" Apple Macbook Pros - 1 PPC and 1 (brand new) Intel running OSX
1 Custom-built workstation w/ dual E5450 3GHz quad-core Intel Xeon processors running Scientific Linux 6
1 old Gateway 486 EISA bus workstation running QNX 4
1 old Dell XPS R450 workstation running Gentoo Linux
1 ARM R&D PC-104 board with I/O and relay daughter boards running Debian Etch Linux
2 Linux-based Buffalo TeraStation NAS units (1TB and 2TB)
1 Linux-based Linksys router

No longer in the house, but still functional and being used daily:

Dell Inspirion Laptop running Ubuntu Linux - gave to grandson a year or two ago
Sony Vaio Laptop running Windows XP Pro - just gave to granddaughter
Dell workstation w/ 3GHz Pentium 4 - gave to daughter about 3 years ago.

I think Toys-R-Us stole their name from us...

[rdyas]

OK, so performance is pretty good. thanks for the confirmation. I would be very insterested if there is anything I can tweak for the write performance, but I understand that raid 5 has to do the parity stuff.

Just fired up the spotify plugin in my squeezebox server (running on the HP Microserver), so I have quality music streaming to anything wirelessly throughout the house

On a more serious note, running backup of server to 2TB usb drive, this was the basis for getting the server with raid - had a disk crash and had to recover a disk at great expense!

thanks again
Roger.

[meton_magis]

Har-de-har-har... That's because I'm an IT and systems engineering consultant. My wife is a physicist. She has 2 17" Apple Macbook Pros, two printers, several firewire attached external discs (including a couple of portable units), and a bunch of other tech toys.

I was considering asking you where you met your wife, but then I read that she has 2 apple computers. Nevermind :P

yeah, I'm a linux zealot, so what??