[SOLVED] SCP: How to Copy Files Recursivly but exclude regex pattern from download?

[miles800]

Hi All,

I want to copy a directory and all of its files using SCP, but exclude a couple of file types. Anyone know of a way to do this? Does SCP support any sort of regex expressions? Hope this is the right forum for this question, I apologize if it is not.

Thank you

[nixnotes]

I found your post because I am looking for an answer to the same thing. rsync has an exclude option so you might consider that. I am going to keep looking, I don't happen by this forum as much these days but if I find something I'll stop back by and let you know.

[patch_linams]

Yesterday I saw nixnotes' reply and the title of the post caught my attention.

Here is a solution I wrote:

#!/bin/sh
# 
# Script name: pscp (pattern-driven recursive secure copy)
# -----------------
#
# Abstract:
# ---------
# This script copies a directory (recursively) from another host excluding
# files which match (a) pattern(s) given by user. 
#
# Description:
# ------------
# It fetches a list of not excluded files in the given directory on another
# host and then copies all the files on the list creating subdirectories
# when needed. In another words, it simulates a pattern-driven recursive 
# secure copying process on top of scp.
#
# Changelog: 
# --------
# Version 0.1 initial release (2007-12-18) 
#
#
# Copyright (C) 2007 patch_linams@yahoo.com
#
# Permission is hereby granted, free of charge, to any person obtaining 
# a copy of this software (the "Software"), to deal in the Software 
# without restriction, including without limitation the rights to use, 
# copy, modify, merge, publish, distribute, sublicense, and/or sell copies 
# of the Software, and to permit persons to whom the Software is furnished 
# to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in 
# all copies or substantial portions of the Software. 
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR "AS IS" AND ANY EXPRESS OR IMPLIED 
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF 
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. 
# IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 
# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES HOWEVER CAUSED AND ON ANY 
# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT ARISING 
# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE.
#

# switches (colors)
RED="\033[31m"
BLU="\033[34m"
END="\033[0m"

# script name
SCRIPT="${0##*/}"

# needed programs
PGS="stty chmod setsid ssh scp mkdir"

# host from which to copy
HOST=""

# user on host
USER=""

# remote directory on host
REMOTE_DIR=""

# patterns (in quotation marks and separated by whitespaces) 
# based on which files will be excluded
PATTERNS=""

# local directory where files will be copied to
DEST="$PWD"

TMP=""
PASS=""

help() {
   cat <<EOF

 $SCRIPT - pattern-driven recursive secure copy

 Usage: $SCRIPT [options ...]

 Where options are:
 -h                           print this text
 -c host                      host from which to copy
 -u user                      user on host
 -d remote_dir                remote directory on host 
 -p 'pattern1 pattern2 ...'   patterns (in quotation marks and separated by 
                              whitespaces) based on which files will be excluded
 -l local_dir                 optional local directory where files will be 
                              copied to [default: current directory]

EOF
   exit 0
}


out() {
    echo -ne "$@"
}

warn() {
    out "\n${RED}$@${END}\n"
}

error() {
    warn "$@"
    help
}

check_programs() {
    for PRG in "$@"; do
	if [ ! -x "$(which "$PRG" | grep bin)" ]; then
	warn "\n \"$PRG\" is needed for the execution of this script!\n
              Please, install \"$PRG\" first."
	exit 1
	fi
    done
}

read_pass() {
    out "Enter password for $USER on $HOST (will not echo): "
    stty -echo
    read PASS
    stty echo
    if [ -z "$PASS" ]; then 
	error "\n No password entered!" 
    fi
}

create_askpass() {
    TMP=$(mktemp /tmp/$USER$HOST.XXXXXXXXXX) || $TMP="/tmp/$USER$HOST.$$"
    if [ -z "$DISPLAY" ]; then 
	DISPLAY=":0.0" 
    fi
    out "#!/bin/sh\n echo $PASS\n#EOF" > $TMP
    chmod 700 $TMP
    SSH_ASKPASS=$TMP
}

check_arg() {
    # $1 - option, $2 -argument value
    case "$2" in
	-*)
            error  " $2 is not a valid argument value for \"-"$1\""!"
	    ;;
    esac
}

remove_opt() {
    for i in "$@"; do
	OPTS=$(echo ${OPTS#"$i"})
    done
}

is_opt_set() {
    # $1 - option, $2 - option value
    if [ -n "$2" ]; then
	error " \"-"$1\"" is already set to \""$2"\"!"
    fi
}

cleanup() {
    $(which rm) -f $TMP 
    stty echo
    out "${END}\n"
    exit 0
}

main() {
    check_programs $PGS

    # read options
    if [ "$#" -eq 0 ]; then 
	help
    fi
    OPTS="$@"
    while getopts ":hc:u:d:p:l:" OPT; do
	case "$OPT" in
	    "h")
	        help
		;;
	    "c" )
		check_arg "$OPT" "$OPTARG"
		is_opt_set "$OPT" "$HOST"
                HOST="$OPTARG"
		remove_opt "-$OPT" "$OPTARG"
		;;
	    "u")
		check_arg "$OPT" "$OPTARG"
		is_opt_set "$OPT" "$USER"
	        USER="$OPTARG"
		remove_opt "-$OPT" "$OPTARG"
		;;
	    "d")
		check_arg "$OPT" "$OPTARG"
		is_opt_set "$OPT" "$REMOTE_DIR"
	        REMOTE_DIR="${OPTARG&#37;/}"
		remove_opt "-$OPT" "$OPTARG"
		;;
	    "p")
	        PATTERNS="$PATTERNS $OPTARG"
		remove_opt "-$OPT" "$OPTARG"
		;;
	    "l")
		check_arg "$OPT" "$OPTARG"
		if [ "$DEST" != "$PWD" ]; then
		    is_opt_set "$OPT" "$DEST"
		fi
	        DEST="$OPTARG"
		remove_opt "-$OPT" "$OPTARG"
		;;
	    "?")
	        error " \"-"$OPTARG"\" is an unknown option!"
		;;
	    ":")
	        error " No argument value for option \"-"$OPTARG\""!"
		;;
	    *)
	        error " Unknown error occurred!"
		;;

	esac
    done
    shift $(($OPTIND - 1))
    if [ -z "$HOST" ]; then
	error " No host specified!"
    elif [ -z "$USER" ]; then
	error " No user specified!"
    elif [ -z "$REMOTE_DIR" ]; then
	error " No remote directory specified!"
    fi
    if [ -n "$OPTS" ]; then 
	error " Invalid options or argument values: \"$OPTS\"!"
    fi

    trap cleanup HUP INT QUIT ABRT TERM EXIT
    read_pass
    if [ $? -eq 0 ]; then
	out "\n${BLU}Copying...${END}\n"
	create_askpass

	if [ -z "$PATTERNS" ]; then
	    setsid scp -r $USER@$HOST:$REMOTE_DIR/* "$DEST/" &>/dev/null
	    exit 0
	fi

	IGNORE=""
	for P in $PATTERNS; do 
	    IGNORE=$IGNORE" -I '"$P"'"
	done

	# get file list
	LIST=$(setsid ssh $USER@$HOST "ls -1 -R $IGNORE $REMOTE_DIR")

	for F in $LIST; do 
	    if [ $(expr "$F" : '/.*:') -gt 0 ]; then
		DIR="${F%:}"
		LDIR="$DEST${DIR#$REMOTE_DIR}"
		mkdir -p $LDIR
	    else
		out "$DIR/$F\n"
		setsid scp $USER@$HOST:"$DIR/$F" "$LDIR/$F" &>/dev/null
	    fi
	done
	out "\n${BLU}Done.${END}"
    fi
}

# execute script with input data
main "$@"
#EOF

And here is a template how to call the script:

pscp -c somehost -u someuser -d /home/someuser/bin -p '*.sh *.bak' -l /home/me/testprogs

That will copy recursively all files in "home/someuser/bin" on "somehost" excluding files matching '*.sh" or "*.bak' to the local directory "/home/me/testprogs" using login "someuser" (a password prompt will appear only once for all files).

Enjoy!

Comments are welcome.

[drl]

Hi.

I have not tried the script by patch_linams, but it looks well-documented and obviously a lot of time was spent on its creation.

I think I would have used tar, which has a lot of exclusionary features, and then transferred the resultant file, and untarred it on the other box.

The rsync command can also do much of that. Neither tar nor rsync are trivial to learn and will likely take some experimentation to learn, but both are well worth the time spent for learning.

Best wishes ... cheers, drl

[nixnotes]

I ended up using rsync. It is infinitely easier (although the script above looks pretty good) and you are able to do exclusions with a simple switch. The only thing I do not like about rsync is that you cannot choose http as a transport for it.

[patch_linams]

The only thing I do not like about rsync is that you cannot choose http as a transport for it.

If you don't need secure transfer there's a very simple way to copy all the files in a directory you want:

wget -m -nH -np -R pattern1,pattern2,... http://host.com/dir1/dir2/

where -R (s. man wget) is

 -A acclist --accept acclist
 -R rejlist --reject rejlist
           Specify comma-separated lists of file name suffixes or patterns to
           accept or reject (@pxref{Types of Files} for more details).

And it's faster than sending data through an encrypted channel.

[halfas]

Hi guys, i looked at this post, and solved a problem i had.

I was runing a linux-explorer script to gather information for analysis on some systems, but the maillog files where too large, the explorer sized around 700 mg. So I edited the linux-explorer script using the info you posted here to exclude the maillog files, thanks a lot!

################################################## ############################
# Systems Log Section
################################################## ############################

Echo "Systems Log Section"

#/bin/cp -R -p /var/log/* ${LOGDIR}/logs

#### Added to exclude maillogs.
cd /var/log

/bin/cp -R $(ls /var/log |grep -v mail) ${LOGDIR}/logs

########################

/bin/dmesg > ${LOGDIR}/logs/dmesg.out