scp / ftp

[linux_2009]

I want to copy 210Gb database from 1 server to other server.

My database consists of different types of files and different directory hierarchy.

I would like to know
1. Which command is better in this scenario?
2. Which is faster command.

Thanks in advance.

[linux_2009]

Also tell me what are the possibilities of copying lasrge files from 1 server to other server.
e.g. i have file with 20g then will it cause any problem for SCP / FTP command?

Thanks in advance.

[D-cat]

I know that SMB (samba) has a built in 4-GB file size limitation which coincides with FAT32 file systems (short-sighted developers!). Since you're not using it, that shouldn't be an issue; I only mentioned it because it's the only protocol I know of that has any file size limitations under Linux.

I don't know what the limitations of built-in Linux versions of other protocols are, if any. FTP also had a 4GB limitation in Windows, but I think that was eradicated with XP-SP2.

FTP has a high overhead which makes it slower, but it's also solid and resumable in the event of a broken transfer. If you're not worried about breaking the connection (e.g. your servers are on the same LAN), then scp should be fine.

If this is something where you might be looking to keep files synchronized between the servers, you might also want to look into rsync. It's no faster on the first run, but subsequent runs can be greatly speed up, even on large files, since only the difference between the files is transferred instead of re-transferring the whole file.

I hope this answers your Q's.

[linux_2009]

Thanks a lot for informative reply.

Then should i consider SCP while copying files?
Also let me know suppose i open 10/20/30 putty terminals and start to copy from every terminal then will it be that as faster as 1 terminal?

What about the bandwidth in this case?

Thanks in advance.

[D-cat]

I'm assuming this is an internal LAN as I write this:
Unless you're running some internal QOS (Quality of Service) for bandwidth limitation, a single execution will limit itself only by the slowest hardware link in the chain (usually the network itself). Running multiple instances would only succeed in increasing hard drive seeks and protocol overhead in the network, both have the potential to slow things down.

If there is a QOS in the link, then the multiple instances should be inverse to the limitation imposed. e.g. say each instance of port 22 is limited to 20% bandwidth, you'll max out bandwidth at 5 instances of scp.

[linux_2009]

My machines are not on LAN.

1 server is at Europe and other is at US.
So i want to know the faster way to copy files.

D-cat says - each instance of port 22 is limited to 20% bandwidth, you'll max out bandwidth at 5 instances of scp.
Please explain this thing in detail.

[D-cat]

Quality of service packages (QoS) are network "shapers" that monitor and regulate the flow of network traffic. Almost all ISP's use QoS in some form, if nothing else, to prevent streaming videos from eating so much bandwidth that other customers can't even load web pages.

Unfortunately, I don't know how your ISP regulates outgoing traffic, so I can't answer how many instances will work best for this transfer. You'll actually have to talk to an engineer in your ISP's Network Operation Center to get this answer the first time, or you can experiment with adding one at a time until bandwidth is full.

You can use something like iftop to monitor your bandwidth usage. (available in SuSE base package, but may still need to be installed through Yast)