[SOLVED] argument list too long (with scp)

[Fermulator]

Consider the following command:

scp hostname:~/sourceDir/*.txt /tmp/destinationDir

This basically says to grab all *.txt files on hostname, in the directory ~/sourceDir, and scp them to the localhost's /tmp/destinationDir.

However, as per "Argument list too long": Beyond Arguments and Limitations | Linux Journal, there is a general problem here that if there are a LOT of files in that sourceDir, we get an error: "Argument list too long"

With the help of Google, and that above referenced page, I was able to figure out a solution. The basic idea here is that we ssh onto the sourceHostname where we want to grab the huge number of files from, run a find, and for every file, scp that file back to the destination hostname. While this command works, it is however REALLY SLOW:

ssh sourceHostname find ~/sourceDir -type f -name '*.txt' -exec scp '{}' destHostname:/tmp/destinationDir/. '\;'

Does anyone have a better solution? I'm trying to couple this method with tar, but it's proving to be elusive...

The idea here is that we can ssh onto the source hostname, run find, pipe into tar, and pipe the tar back on the localhost through the terminal, and untar at the same time.

Something like: (SYNTAX INCORRECT)

ssh sourceHostname 'find ~/sourceDir -type f -name "*.txt" | tar cz --files-from -' | tar xzvf - -C /tmp/destinationDir

[nmset]

The ssh utility won't transfer the file to anywhere. You'll have to do the job in 2 steps :

Create the tar file,
then transfer it with scp or rsync.

ssh sourceHostname 'find ~/sourceDir -type f -name "*.txt" | tar czf archive.tar.gz --files-from -'
rsync -aP sourceHostname:/path/to/archive.tar.gz /dest

This 'Too many arguments' error is new to me, I'm happy to have learned it !

[Fermulator]

Interesting, if possible, I'd prefer to not have to do it in two steps (creating a file, then going back and extracting it will take double the time)

This post says you can do it:
macosx.com - View Single Post - Piping Tar datastream over SSH
That poster proposes something like:

ssh remote "cd /usr/local/stuff; tar cf - ." | tar xf -

If we translate this to what I'm trying to do:

ssh sourceHostname "cd ~/sourceDir; find . -type f -name \"*.txt\" | tar czv --files-from -" | tar xzvf -

I'm EXPECTING from the above command, that we're:
1) sshing to sourceHostname
2) chdir to the sourceDir (to avoid relative paths in the tar operation)
3) find all files in the sourceDir, and pipe to tar
4) tar is supposed to output to stdout (over the ssh terminal), and that final pipe (outside of the ssh command), will take the stdout, and extract that tar data to the current directory on destinationHost.

But, it gives the error:
tar: Failed to open '/dev/sa0': Operation not supported

[nmset]

ssh sourceHostname "cd ~/sourceDir; find . -type f -name \"*.txt\" | tar czv --files-from -" > file.tar.gz

seems to redirect stdout to a named archive on destination host, which we can easily unpack. Trying to untar the stdout on the fly resulted constantly with partial retrieval and error messages

tar: . : la fonction utime a échoué: Opération non permise

which seems to translate to

tar:.:utime function failed: Operation not allowed

[nmset]

(creating a file, then going back and extracting it will take double the time)

Two connections for sure, twice the time not sure, for in the first connection, the archive is not transferred. You may even type the password once only if you use public key authentication through ssh-agent.

[Fermulator]

Figured it out!

ssh sourcehostname 'cd ~/sourceDir; find . -type f -name "*.txt" | tar czvf - --files-from -' | tar zxvf - -C destinationDir

[Fermulator]

Also, this page was a great help for this purpose:

Pushing & Pulling Files Around Using tar, ssh, scp, & rsync Lâmôlabs