Find the answer to your Linux question:
Results 1 to 6 of 6
ok, i have this script that will copy files from one directory to another but will ask the user how many files to copy simultaneously (this is where pipe comes ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Linux Enthusiast
    Join Date
    Jun 2005
    Location
    Odessa, FL
    Posts
    586

    BASH script pipe problem


    ok, i have this script that will copy files from one directory to another but will ask the user how many files to copy simultaneously (this is where pipe comes in). a portion of the script is as follows (the portion I have a question about):
    Code:
    	count=1
    	while [ "${DL[$count]}" != "" ]; do
    		clear
    
    		counter=1
    		echo "Currently copying: \c"
    			echo -e "${DL[$count]}\c"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			echo -e " and ${DL[$((counter+count))]}\c"
    			counter=$((counter+1))
    		done
    		echo ""
    
    		counter=1
    		runstr="cp $from${DL[$count]} $to"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			runstr="$runstr | cp $from${DL[$((counter+count))]} $to"
    			counter=$((counter+1))
    		done
    		$runstr
    		count=$((count+simul))
    	done
    the array "{DL[$x]}" is for storing the files to be downloaded/copied
    "$from" is the directory the files are copied from, by user input (in this case, /home/josh/temp/)
    "$to" is the directory the files are copied to, by user input (in this case, /home/josh/temp1/)
    "$simul" is the user input...the number of files to be copied simultaneously

    I used the pipe command to copy multiple files (the number specified by the user simultaneously) but get a weird error when I execute it (this time with simul=10):
    Code:
    Currently copying: temp1 and temp10 and temp11 and temp12 and temp13 and temp14 and temp15 and temp16 and temp17 and temp18
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    cp: omitting directory `/home/josh/temp1/'
    cp: cannot stat `|': No such file or directory
    cp: cannot stat `cp': No such file or directory
    i had 100 "test" files in each directory to see how this script works and I showed the output of what happened with just the first 10 (to keep it short). this continues all the way until it's done copying them all.

    can someone please help me resolve this?

    Thanks in advance!!

  2. #2
    Linux User muha's Avatar
    Join Date
    Jan 2006
    Posts
    290
    it's your script right? Please post the whole thing
    How are you executing the script?

    Why not use? Or do you want to specify each outputdirectory?
    cp <file1> <file2> <file3> <directory>
    Copy multiple files into a directory

    Why are you storing the copy commands, and not executing them straight-away in the while loop? Or is that not an option?

    What's the output when you do: echo $runstr
    Before executing (after the while loop), look at it, maybe you can find out what's going wrong ...
    Now what? You have Linux installed and running. The GUI is working fine, but you are getting tired of changing your desktop themes. You keep seeing this "terminal" thing. Don't worry, they'll show you what to do @
    <~ http://www.linuxcommand.org/ ~>

  3. #3
    Linux Enthusiast
    Join Date
    Jun 2005
    Location
    Odessa, FL
    Posts
    586
    Quote Originally Posted by muha
    it's your script right? Please post the whole thing
    How are you executing the script?

    Why not use? Or do you want to specify each outputdirectory?
    cp <file1> <file2> <file3> <directory>
    Copy multiple files into a directory

    Why are you storing the copy commands, and not executing them straight-away in the while loop? Or is that not an option?

    What's the output when you do: echo $runstr
    Before executing (after the while loop), look at it, maybe you can find out what's going wrong ...
    the output of "echo $runstr" is:
    Code:
    cp /home/josh/temp/temp1 /home/josh/temp1/ | cp /home/josh/temp/temp10 /home/josh/temp1/ | cp /home/josh/temp/temp11 /home/josh/temp1/ | cp /home/josh/temp/temp12 /home/josh/temp1/ | cp /home/josh/temp/temp13 /home/josh/temp1/ | cp /home/josh/temp/temp14 /home/josh/temp1/ | cp /home/josh/temp/temp15 /home/josh/temp1/ | cp /home/josh/temp/temp16 /home/josh/temp1/ | cp /home/josh/temp/temp17 /home/josh/temp1/ | cp /home/josh/temp/temp18 /home/josh/temp1/
    this seems right to me...
    is there a limit to how many pipes you can have going at a time?
    i know that "cp /home/josh/temp/temp1/ /home/josh/temp1/ | cp /home/josh/temp/temp2/ /home/josh/temp1/" works...
    i also ran the script with only 2 simultaneous copies and it still gave me the same thing, so i'm sure it's the script somewhere...

    the whole script:
    Code:
    # !/bin/bash
    # syncs any folders but especially designed for music databases
    
    #STATING SPECIAL CHARACTER STYLES
    BOLD="\033[1m"
    RED="\E[1;31m"
    END="\033[0m"
    
    #STATING MODULES
    main()
    {
    	counting=1
    	while [ "${FROM[$counting]}" != "" -o "${TO[$counting]}" != "" ]; do
    		unset FROM[*]
    		unset TO[*]
    		unset DL[*]
    	done
    
    	ls -1 $from > ~/.sync/from
    	ls -1 $to > ~/.sync/to
    
    	exec 6<&0
    	exec < ~/.sync/from
    	read fromstr
    	counting=1
    	while [ "$fromstr" != "" ]; do
    		FROM[$counting]="$fromstr"
    		high=$counting
    		counting=$((counting+1))
    		read fromstr
    	done
    	exec 0<&6 6<&-
    
    	exec 6<&0
    	exec < ~/.sync/to
    	read tostr
    	counting=1
    	while [ "$tostr" != "" ]; do
    		TO[$counting]="$tostr"
    		counting=$((counting+1))
    		read tostr
    	done
    	exec 0<&6 6<&-
    	echo -e $BOLD"Arrays created"$END
    	sleep 1
    }
    
    getfirst()
    {
    	echo $1
    }
    
    run()
    {
    	clear
    
    	main
    	count=1
    	dlcount=1
    	while [ "${FROM[$count]}" != "" ]; do
    		clear
    		if [ $count -lt $high ]; then
    			div=$((count*100))
    			echo -e $BOLD"Determining which files need to be synced...$END $((div/high))%"
    		else
    			echo -e $BOLD"Determining which files need to be synced...$END 100%"
    			sleep 1
    		fi
    		found=no
    		counter=1
    		while [ "${TO[$counter]}" != "" ]; do
    			if [ "${TO[$counter]}" = "${FROM[$count]}" ]; then
    				found=yes
    				fromsize="${FROM[$count]}"
    				tosize="${TO[$counter]}"
    				counter=0
    			else
    			counter=$((counter+1))
    			fi
    		done
    		if [ "$fromsize" = "$tosize" -a $(getfirst $(du $to$tosize)) -ne $(getfirst $(du $from$fromsize)) ]; then
    			found=no
    		fi
    		if [ "$found" = "no" ]; then
    			DL[$dlcount]="${FROM[$count]}"
    			dlcount=$((dlcount+1))
    		fi
    		count=$((count+1))
    	done
    	
    	echo ""
    	count=1
    	if [ "${DL[1]}" = "" ]; then
    		echo -e $BOLD"No files to sync...exiting"$END
    		exit
    	else
    		echo -e $BOLD"These files will be downloaded:$END \c"
    	fi
    	while [ $count -lt $((dlcount-1)) ]; do
    		echo -e "${DL[$count]}, \c"
    		count=$((count+1))
    	done
    	echo -e "${DL[$count]}"
    	echo ""
    	echo "Press [ENTER] to continue..."
    	read
    	
    ##############################################
    	count=1
    	while [ "${DL[$count]}" != "" ]; do
    		clear
    		if [ $count -lt $high ]; then
    			div=$((count*100))
    			echo -e $BOLD"Syncing files...$END $((div/high))%"
    		else
    			echo -e $BOLD"Syncing files...$END 100%"
    		fi
    		echo ""
    
    		counter=1
    		echo -e $BOLD"Currently copying:$END \c"
    			echo -e "${DL[$count]}\c"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			echo -e " and ${DL[$((counter+count))]}\c"
    			counter=$((counter+1))
    		done
    		echo ""
    
    		counter=1
    		runstr="cp $from${DL[$count]} $to"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			runstr="$runstr | cp $from${DL[$((counter+count))]} $to"
    			counter=$((counter+1))
    		done
    		$runstr
    		count=$((count+simul))
    	done
    ##############################################
    
    	echo "DONE!"
    }
    
    #ACTUAL PROGRAM EXECUTION
    if [ -d ~/.sync ]; then
    	echo "~/.sync exists...continuing"
    else
    	echo "~/.sync does not exist...creating"
    	mkdir ~/.sync
    fi
    sleep 1
    
    clear
    
    if [ "$1" = "other" ]; then
    	echo -n "Enter the directory to sync from (include the / at the end of the directory): "
    	read from
    	if [ -d $from ]; then
    		shift
    		else
    		echo -e $RED"Error: $from is an invalid directory!"$END
    		echo -e $RED"EXITING"$END
    		exit
    	fi
    
    	echo -n "Enter the directory to sync to (include the / at the end of the directory): "
    	read to
    	if [ -d $to ]; then
    		shift
    		else
    		echo -e $RED"Error: $to is an invalid directory!"$END
    		echo -e $RED"EXITING"$END
    		exit
    	fi
    elif [ "$1" = "music" ]; then
    	sudo mount /network/from/music/
    	from="/network/alex/music/"
    	to="/network/server/shared/Music/"
    else
    	echo -e $RED"Error: Either you entered an invalid option or you didn't enter an option after the command"$END
    	echo "sync music = sync server's music with alex's computer"
    	echo "sync other = sync any directories specified by the user"
    	exit
    fi
    echo "Syncing files from $from to $to"
    echo "How many files would you like to copy simultaneously? "
    read simul
    
    run
    i originally wrote this script to sync files from one music directory to another. it would copy the files if the "from" music directory contained a file that the "to" one didn't have, or if they both had the file but the "from" directory had files that were slightly different in size from the "to" directory's files.

    the script works perfectly if i manually put in how many simultaneous copies there will be (ie: "cp $from${DL[$count]} /home/josh/temp1/ | cp $from${DL[$((count+1))} /home/josh/temp1/"

    i wrote this script, rather than use rsync, because rsync seems to take way too long to do what I want. This script doesn't have the extra functionality (recursion, etc...) but is MUCH faster. I ran rsync across these directories (820 files on one, 796 on the other) and it took over 3 hours. With the script I wrote, specifically for this case, it takes about 40 seconds of processing which files should be copied...the rest is just copying them over (which, with only a difference of 24 songs over a LAN, it shouldn't take long at all).

    So, can someone please help me? The section broken off with all of the "#" in a row is the section I need help the most with. The rest of the script works very well.

    Thanks in advance!!

  4. #4
    Linux User muha's Avatar
    Join Date
    Jan 2006
    Posts
    290
    hmm? temp1 or temp1/ ?
    you wrote:
    i know that:
    Code:
    cp /home/josh/temp/temp1/ /home/josh/temp1/
    but temp1 is a file right? So loose the / in the first temp1
    I don't understand what copy you want to make. Maybe you can differentiate between file1 and dir1. Do you want something like this :?
    Code:
    cp /home/josh/temp/file1 /home/josh/dir1/ | cp /home/josh/temp/file2 /home/josh/dir2/ | etc ...
    Now what? You have Linux installed and running. The GUI is working fine, but you are getting tired of changing your desktop themes. You keep seeing this "terminal" thing. Don't worry, they'll show you what to do @
    <~ http://www.linuxcommand.org/ ~>

  5. #5
    Linux Enthusiast
    Join Date
    Jun 2005
    Location
    Odessa, FL
    Posts
    586
    ok, i sort of figured it out, but it still came up with some errors (but worked as i wanted it to).

    i changed this:
    Code:
    		counter=1
    		runstr="cp $from${DL[$count]} $to"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			runstr="$runstr | cp $from${DL[$((counter+count))]} $to"
    			counter=$((counter+1))
    		done
    		$runstr
    		count=$((count+simul))
    to...
    Code:
    		counter=1
    		runstr="$(cp --target-directory=$to $from${DL[$count]})"
    		while [ $counter -lt $simul -a "${DL[$counter+count]}" != "" ]; do
    			runstr="$runstr | $(cp --target-directory=$to $from${DL[$((counter+count))]})"
    			counter=$((counter+1))
    		done
    		$runstr
    		count=$((count+simul))
    but since errors were still thrown (like "command '\' not found") i decided to drop the whole simultaneous-copy support. it just sounded like an interesting concept to implement but it's no big deal at all. i would only copy 10-15 new files at a time so a simultaneous copy feature didn't seem all that practical, i was just learning/trying to figure out how i would combine commands using pipes (with user input) in case i wanted to use it again in the future.

    thanks for your help, muha

  6. #6
    Linux User muha's Avatar
    Join Date
    Jan 2006
    Posts
    290
    yeah, i see.
    I think you almost got it. Just always look at your commands output carefully ..
    cyas!
    Now what? You have Linux installed and running. The GUI is working fine, but you are getting tired of changing your desktop themes. You keep seeing this "terminal" thing. Don't worry, they'll show you what to do @
    <~ http://www.linuxcommand.org/ ~>

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •