ZFS, anyone?

[Lakshmipathi]

Hey colleague!


Well, performance testing, I guess that's true in a sense. Actually what I did was I divided a drive in four equal partitions with these filesystems: rieser, etx3, jfs and zfs-fuse.

I then used dd to create huge (4GB) files on them, copied these files and deleted them.
Then I created tens of thousands of small files on the filesystems, copied those around and deleted them.

And I timed each step. But since we're on the subject, I am curious. Is this methodology any use for comparing filesystem performance?

I think hardware setup (like RAM,CPU) and status of machines (does any other application is running on that machine or not) is crucial to get some-what accurate picture .

Yes , dd tool is used to simple testing . tools like iozone and dbench are widely used.
For instance,iozone provides details about read/re-read ,stride read (which is like - Move to N of set and reading from it and then move to 2*N often and read and move to 3*N and so on )
write/re-write etc.

So these tools can be used with different file system running on similar hardware configuration
get the numbers and compare them and select which file system might be better for our application. (For example , if you have lot of movies , then you select file system which provides
better read performance because you won't care about write performance )

Remember , I'm pretty new to this testing field too, these thought can be completely wrong too .
and thanks Freston, for the below script - if needed , I think i can modify it for suit my requirement and use it .

#!/bin/bash                                                                                                                             

COUNT1="0"                          
NRSMALLFILES="10000"                

# These arrays correlate in COUNT1
PARTITION=(     hdb1              
                hdb2              
                hdb3              
                hdb4              
          )                       

FILESYSTEM=(    reiser 
                ext3   
                jfs    
                zfs    
           )           


# Don't run this at home!
exit

#########################
BigZero () {             

echo ""
echo "##############################################"
echo "Testing on ${PARTITION[$COUNT1]} formatted ${FILESYSTEM[$COUNT1]}"
echo ""                                                                 
echo "Create bigfile on ${FILESYSTEM[$COUNT1]}"                         
        dd      bs=4M \                                                 
                count=1000 \                                            
                if=/dev/zero \                                          
                of=/mnt/$dir/bigfile                                    
echo ""                                                                 
echo "Copy bigfile on ${FILESYSTEM[$COUNT1]}"                           
        time cp         /mnt/$dir/bigfile \                             
                        /mnt/$dir/copiedbigfile                         

echo ""
echo "Delete big files on ${PARTITION[$COUNT1]} formatted ${FILESYSTEM[$COUNT1]}"
        time rm         /mnt/$dir/{bigfile,copiedbigfile}                        

}



#########################
CreateSmallFiles () {    
COUNT2="0"               
mkdir /mnt/$dir/dirwithsmallfiles
echo -n "Creating $NRSMALLFILES small files "

while [ $COUNT2 -lt $NRSMALLFILES ] ; do
        dd      bs=512 \                
                count=1 \               
                if=/dev/zero \          
                of=/mnt/$dir/smallfile$COUNT2 \
                2> /dev/null                   
        ((COUNT2++))                           
done                                           

}



#########################
CopyAndCleanup () {      

echo ""
echo "done... now copying small files"
        time cp         /mnt/$dir/smallfile* \
                        /mnt/$dir/dirwithsmallfiles

echo ""
echo "Cleaning up!"
echo -n "Deleting small files takes: "
        time rm         /mnt/$dir/smallfile*

echo ""
echo -n "Deleting whole directory with small files takes: "
        time rm -rf     /mnt/$dir/dirwithsmallfiles


}




#########################
#
#    Work work work...
#
#########################
for dir in ${PARTITION[*]} ; do

        BigZero
        time CreateSmallFiles
        CopyAndCleanup

# Increment count for dir and arrays
((COUNT1++))

done

[Freston]

and thanks Freston, for the below script - if needed , I think i can modify it for suit my requirement and use it

Sure, go ahead. Just remember it doesn't come with warranties, implied or expressed, for reliability, merchantability (whatever that is), or even fitness for a particular purpose

I think hardware setup (like RAM,CPU) and status of machines (does any other application is running on that machine or not) is crucial to get some-what accurate picture .

1.4Ghz of raw force, upped the RAM to 1GB and no, it wasn't doing anything useful. It looks like it's hurt, lying half open with a (extra) disk coming out.
I ran the tests a couple of times, because I'm more interested in trends than in individual measurements.

Yes , dd tool is used to simple testing . tools like iozone and dbench are widely used.
For instance,iozone provides details about read/re-read ,stride read (which is like - Move to N of set and reading from it and then move to 2*N often and read and move to 3*N and so on )
write/re-write etc.

Ok, interesting. I found iostat on my system. I'll give that a look as well. It's not of major importance anymore, because I've been able to observe HROAdmin26's claim that performance was lacking. By the way, I found on the mailing list comparable results as what I got. And I also found something there about dedub being in testing. I should read up on that.

Maybe this zfs-fuse thing isn't for Linux as a primary filesystem, but thus far I'm glad I'm giving it a spin, if only for compatibility reasons.

So these tools can be used with different file system running on similar hardware configuration
get the numbers and compare them and select which file system might be better for our application. (For example , if you have lot of movies , then you select file system which provides
better read performance because you won't care about write performance )

Remember , I'm pretty new to this testing field too, these thought can be completely wrong too .

Why, no... it makes sense what you say, at least logically. Dunno much about the Real World though
Joke of course, I mean large HUGE clusters of storage arrays. I don't know how that would respond, and I don't know much about distributed filesystems other than the occasional nfs, Samba, sshfs and vboxvfs... but I have no measurements other than copying a DVD from here to there and pushing the task to the background because it's taking to long

[Lakshmipathi]

Sure, go ahead. Just remember it doesn't come with warranties, implied or expressed, for reliability, merchantability (whatever that is), or even fitness for a particular purpose

Ok, interesting. I found iostat on my system. I'll give that a look as well. It's not of major importance anymore, because I've been able to observe HROAdmin26's claim that performance was lacking. By the way, I found on the mailing list comparable results as what I got. And I also found something there about dedub being in testing. I should read up on that.

Maybe this zfs-fuse thing isn't for Linux as a primary filesystem, but thus far I'm glad I'm giving it a spin, if only for compatibility reasons.

Yes,I heard about iostat as a very useful tool but haven't used them yet.
I agree - zfs-fuse is not for primary file system - just adds more feature or new flavour to linux file system.

Why, no... it makes sense what you say, at least logically. Dunno much about the Real World though
Joke of course, I mean large HUGE clusters of storage arrays. I don't know how that would respond, and I don't know much about distributed filesystems other than the occasional nfs, Samba, sshfs and vboxvfs... but I have no measurements other than copying a DVD from here to there and pushing the task to the background because it's taking to long

Network bandwidth plays vital role while evaluating Distributed file systems ...but over all i would say performance testing will help us understand file systems in better way.

[HROAdmin26]

It's not even close to an apples-to-apples comparison for you, but I cleaned up your script for a Sparc Solaris 10 system and ran it against a ZFS volume.

The ZFS volume is on a 200GB LUN carved from a SAN-attached Hitachi AMS200 array.

Script results:

##############################################
Testing on HDS_200GB/Data formatted zfs

Create bigfile on zfs
1000+0 records in
1000+0 records out

Copy bigfile on zfs

real    1m32.844s
user    0m0.007s
sys     0m19.882s

Delete big files on HDS_200GB/Data formatted zfs

real    0m0.214s
user    0m0.001s
sys     0m0.169s
Creating 10000 small files 
real    0m42.765s
user    0m11.341s
sys     0m34.282s

done... now copying small files

real    0m2.916s
user    0m0.247s
sys     0m2.639s

Cleaning up!
Deleting small files takes: 
real    0m1.122s
user    0m0.096s
sys     0m0.873s

Deleting whole directory with small files takes: 
real    0m0.570s
user    0m0.051s
sys     0m0.516s

ZFS Details:

zfs get all HDS_200GB/Data 
NAME            PROPERTY       VALUE                  SOURCE
HDS_200GB/Data  type           filesystem             -
HDS_200GB/Data  creation       Thu May 13 16:06 2010  -
HDS_200GB/Data  used           358K                   -
HDS_200GB/Data  available      196G                   -
HDS_200GB/Data  referenced     358K                   -
HDS_200GB/Data  compressratio  1.00x                  -
HDS_200GB/Data  mounted        yes                    -
HDS_200GB/Data  quota          none                   default
HDS_200GB/Data  reservation    none                   default
HDS_200GB/Data  recordsize     128K                   default
HDS_200GB/Data  mountpoint     /HDS_200GB/Data        default
HDS_200GB/Data  sharenfs       off                    default
HDS_200GB/Data  checksum       on                     default
HDS_200GB/Data  compression    off                    default
HDS_200GB/Data  atime          on                     default
HDS_200GB/Data  devices        on                     default
HDS_200GB/Data  exec           on                     default
HDS_200GB/Data  setuid         on                     default
HDS_200GB/Data  readonly       off                    default
HDS_200GB/Data  zoned          off                    default
HDS_200GB/Data  snapdir        hidden                 default
HDS_200GB/Data  aclmode        groupmask              default
HDS_200GB/Data  aclinherit     secure                 default
HDS_200GB/Data  canmount       on                     default
HDS_200GB/Data  shareiscsi     off                    default
HDS_200GB/Data  xattr          on                     default

Script as run:

#!/bin/bash 

COUNT1="0"                          
NRSMALLFILES="10000"                

# These arrays correlate in COUNT1
PARTITION=(     HDS_200GB/Data              
          )                       

FILESYSTEM=(    zfs
           )           


# Don't run this at home!

#########################
BigZero () {             

echo ""
echo "##############################################"
echo "Testing on ${PARTITION[$COUNT1]} formatted ${FILESYSTEM[$COUNT1]}"
echo ""                                                                 
echo "Create bigfile on ${FILESYSTEM[$COUNT1]}"                         
        dd bs=4096000 count=1000 if=/dev/zero of=/$dir/bigfile 
echo ""                                                                 
echo "Copy bigfile on ${FILESYSTEM[$COUNT1]}"                           
        time cp /$dir/bigfile /$dir/copiedbigfile

echo ""
echo "Delete big files on ${PARTITION[$COUNT1]} formatted ${FILESYSTEM[$COUNT1]}"
        time rm /$dir/{bigfile,copiedbigfile}                        

}



#########################
CreateSmallFiles () {    
COUNT2="0"               
mkdir /$dir/dirwithsmallfiles
echo -n "Creating $NRSMALLFILES small files "

while [ $COUNT2 -lt $NRSMALLFILES ] ; do
        dd bs=512 count=1 if=/dev/zero of=/$dir/smallfile$COUNT2 2> /dev/null
        ((COUNT2++))                           
done                                           

}



#########################
CopyAndCleanup () {      

echo ""
echo "done... now copying small files"
        time cp /$dir/smallfile* /$dir/dirwithsmallfiles

echo ""
echo "Cleaning up!"
echo -n "Deleting small files takes: "
        time rm /$dir/smallfile*

echo ""
echo -n "Deleting whole directory with small files takes: "
        time rm -rf /$dir/dirwithsmallfiles


}




#########################
#
#    Work work work...
#
#########################
for dir in ${PARTITION[*]} ; do

        BigZero
        time CreateSmallFiles
        CopyAndCleanup

# Increment count for dir and arrays
((COUNT1++))

done

exit 0

Sparc Details:

Memory: 4096 Megabytes
CPU Type: 1503 MHz SUNW,UltraSPARC-IIIi
Number of CPU's: 2
System Configuration: Sun Microsystems sun4u Sun Fire V240
Kernel Version: 5.10 Generic_127111-11

[Freston]

Hey HROAdmin26, thanks! Good to see a comparison.

It's a bit weird though, because my old x86 iron with zfs-fuse is in some respects faster than your Sparc running ZFS natively. Well, it's difficult to compare of course. Could be any of several things.

To bad your output misses the transfer rate from `dd` on BigFile... dd outputs this on stderr so I had to do this:
sh script>outputfile 2>&1

to get this:

##############################################
Testing on hdb4 formatted zfs                 

Create bigfile on zfs
1000+0 records in    
1000+0 records out   
4194304000 bytes (4.2 GB) copied, 222.18 s, 18.9 MB/s

Copy bigfile on zfs

real    1m54.293s
user    0m1.731s 
sys     0m5.646s 

Delete big files on hdb4 formatted zfs

real    0m3.638s
user    0m0.001s
sys     0m0.097s
Creating 10000 small files 
real    0m36.694s          
user    0m13.253s          
sys     0m17.070s          

done... now copying small files

real    0m10.841s
user    0m0.282s
sys     0m1.416s

Cleaning up!
Deleting small files takes:
real    0m2.053s
user    0m0.065s
sys     0m0.191s

Deleting whole directory with small files takes:
real    0m1.954s
user    0m0.015s
sys     0m0.209s

With your experience, you can probably say that running ZFS natively will be much quicker than my 18.9 MB/s on zfs-fuse?

Network bandwidth plays vital role while evaluating Distributed file systems ...but over all i would say performance testing will help us understand file systems in better way.

I never given it much thought. And that always somehow nagged me a little. But how do you start getting familiar with something you where previously unfamiliar with? Oh, I dunno? Throw some tests at it? Sure! it would be much more useful in the short term to read up on the new and *cough* improved event viewer for Server2008... but this is much more interesting.

But I think I'll let it rest for a while at least. My new server came in A GuruPlug...

...once that is installed properly, my old server I can use for target practice for more extensive testing. Both zfs-fuse and my other little projects. But I found why they call it a GuruPlug; it doesn't bend to my will as easily as standard x86 hardware (getting U-Boot to load a homebrew image off NAND is higher magic) but it'll work. It'll work...