Results 1 to 3 of 3
Thread: copy a large amount of files.
Enjoy an ad free experience by logging in. Not a member yet? Register.
- Join Date
- Aug 2012
copy a large amount of files.
I have a NAS on my network that I need to copy files from into my new Ubuntu server, the total size is 2.3 TB and there are a lot of small files. I am a little unsure when it comes to the right tool to use here, the file's integrity is very important because I cant afford data to be corrupted ( its a medical office, and I am copying medical records). Rsync is very slow because there are a lot of directories and little files, I would use netcat in conjunction with tar but I cant install it on the NAS, and there isnt enough space to make a tar ball on the nas.
Hi and welcome.
On a filelevel, rsync is argueably the best tool for the job: mature, stable, featurerich, etc.
rsync appears slow, because two filelists have to be built and compared before an actual copy happens.
This takes time as there are a myriad of stat() calls.
From your post, this copyjob seems to be a one-time task.
So one might think of cp or mv instead of rsync.
Both are possible, but only rsync offers the possibility to continue if the transfer should be interrupted somehow.
Also, a second rsync run can be used to verify integrity and/or pick up latest additions to that NAS.
Do you have the chance to mount the filesystem on the NAS (and ubuntu server) with -noatime?
This will help at least a bit to speed it up.
Another approach is to copy on device/partition level.
a) Do you have root shell access on that NAS?
b) Does it have a filesystem, that ubuntu can also use? Like ext3, ext4, xfs?
c) Are you able to identify and unmount the data partition?
d) Is 1) dd and netcat or 2) partimage available on the NAS?
If the answers to a)-d) are yes, then you could create a partition on the ubuntu server that is at least as big as the NAS data partition.
Then login to the NAS, shutdown all services accessing the datapartition and unmount the datapartition.
The rest is a image copy over the network to that new ubuntu partition.
This should be pretty fast, depending on the specs of NAS, ubuntu server and network I would expect at least 30MByte/s, probably more.
Once the image has been transfered to the ubuntu partition, it might be a good idea to fsck it first.
Then mount it.You must always face the curtain with a bow.
- Join Date
- Aug 2012
hey thanks for the response, I think ill end up just using rsync, it seems like the most straight forward thing to do, and I definitely need to verify it once its done, How would you continue if the transfer is interrupted? and should I use -c (compress) when I transfer the files ? the command I used is
rsync -avPc would that be correct?