backing up 300,000 files

[cgi-adewolf]

Hello all:
Long time linux user here.
I started a new job and they have a Linux (CentOS 5.3 32bit) box with one disk (looks like it's a RAID 1) for everything that has over 300,000 files in one directory. I am working on backups (backup to local disk using dump and copy dump files to remote server) and there is no other disk on another controller available. If I try to count the number of files (ls */*/*/*|wc -l) it chokes and errors out. Any suggestions?

Alex DeWolf
San Diego, California

[Irithori]

find /DIRECTORY -type f |wc -l

will count the files.

On backup:
Depends, how big you plan.
Is rsync enough?
For a network wide backup I usually recommend bacula.

[cgi-adewolf]

Got that. I am not sure the best way to back this up with the least I/O impact on the server. This is production.

[Irithori]

300000 doesnt sound too scary.
My biggest number of files per server is 40million (although in a directory structure, not all in one dir)

[cgi-adewolf]

How did you back it up? Was it a production server where I/O was an issue? Did you use an external (USB) device (disk drive or tape drive)?

[HROAdmin26]

When you get into the millions-of-files volume, snapshots and "raw disk" (imaging) backups during off hours will get the best throughput.

The storage location is irrelevant as long as it can write at the speeds you require.

[Irithori]

When you get into the millions-of-files volume, snapshots and "raw disk" (imaging) backups during off hours will get the best throughput.

That is of course correct.

But dependend on your hardware´s capabilities you might be able to implement "regular" backup as well.
A raid 10 over 24x 146GByte 15k harddiscs with a decent controller is surely more performant than a raid5 with 3x 750Gbyte 7.2k and a consumer grade controller.

So -in my case- we opted for the "kill the problem with hardware" option,
just to have a consistent way of backuping our network.
Yes, this is a (one of two redundant actually) production server
and yes, it is backuped during operation.


Bottom line:
If your hardware can take it, rsync or better bacula are viable options for backing up these 300000 files

[cgi-adewolf]

These machines are in a colocation and have only one disk controller each, so the one pipe analogy applies here. If I ran the hardware I would make sure there was 2 RAID controllers and one fibre controller each.
I have been checkinto rsync, I will check into bacula.

Thanks

[Rubberman]

When you get into the millions-of-files volume, snapshots and "raw disk" (imaging) backups during off hours will get the best throughput.

The storage location is irrelevant as long as it can write at the speeds you require.

File system types are also important here. Some choke with too many files. Others are OK with large numbers of files in a single directory.