Welcome to Linux Forums!

With a comprehensive Linux Forum, information on various types of Linux software and many Linux Reviews articles, we have all the knowledge you need a click away, or accessible via our knowledgeable members.

Linux Forum ArticlesLinux ForumsLinux Forum DownloadsLinux HostsFree MagazinesJobs
Home|Register|FAQ|Member List|Calendar|Unanswered Posts|Forum Rules|Today's Posts|Advanced Search|
SEARCH FOR IN
Go Back   Linux Forums > GNU Linux Zone > Linux Programming & Scripting
Reload this Page Help. noob needs help with file copy!
Linux Forums
Linux Forums
Welcome To The Linux Forums!
Welcome to Linux Forums. We pride ourselves in being one of the largest Linux communities on the web, we encourage you to REGISTER on our forums and participate in the community. There are over 150,000 members ready to answer your questions. JOINING US today will allow you to make new posts, get support, send messages to other members and submit downloads to our downloads directory and many other great features!

Linux Programming & Scripting C, Perl, PHP, Bash Scripts, anything programming or script related post in here!

Reply
 
Thread Tools Display Modes
Old 07-16-2008   #1 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Help. noob needs help with file copy!

So I have a very simple question that has come to be a huge headache.

I need to copy files from FolderA into FolderB replacing all files that already exist in FolderB UNLESS the date of the file in FolderB is newer than the date from the same file in FolderA. In the case that file is newer in the destination, I need to skip copy of that file and move on.

I have tried so many different things and nothing has been able to do what I need.

I have tried a simple cp -i but i need to filter by date and choose "don't overwrite" if file being copied is older, but owverwite is file date is same.

there is a long story to go along with this as well.

Thanks in advance
wildside is offline   Reply With Quote
Old 07-16-2008   #2 (permalink)
Just Joined!
 
Join Date: Sep 2007
Location: Lafayette, IN
Posts: 51
Sounds like what you need is the rsync command:

rsync -a /path/to/source /path/to/destination

See the rsync man page for more options. rsync is a very powerful utility.
Ben Cotton is offline   Reply With Quote
Old 07-16-2008   #3 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Thanks. that is one I have not man'd yet. I will check it out.
wildside is offline   Reply With Quote
Old 07-17-2008   #4 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Thank you for the rsync pointer.

So rsync looks like what i need. I ran some "dry-runs" with it and also some test runs.
I dont think -a (append) is what i want since that would write data to the end of the file and not replace the whole file.

using this convention:
$ rsync -vru --progress FolderA/ FolderB/

It only copies files from FolderA that are newer than files in FolderB.
It skips files that are the same size or date.
This is an expect from the rsync man page:

-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file's, it will be updated if the sizes are different.)


So what i need to do is force rsync to use update to copy a file if it is equal or newer. NOT if it is ONLY newer.


Any ideas here?
wildside is offline   Reply With Quote
Old 07-17-2008   #5 (permalink)
Bigtomrodinator
 
bigtomrodney's Avatar
 
Join Date: Nov 2004
Location: Sunny South-East of Ireland
Posts: 5,185
Not to avoid answering your question, but surely if the date is equal surely it doesn't need to update?

rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?
__________________
Registered Linux user #378740
New members read here / Forum Rules
#linuxforums on irc.freenode.net
bigtomrodney is online now   Reply With Quote
Old 07-17-2008   #6 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Quote:
Originally Posted by bigtomrodney View Post
Not to avoid answering your question, but surely if the date is equal surely it doesn't need to update?

rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?
Not so. Now for the long story.

Ok a really short version of the long story.

We have a San Server with 43TB of user data. We backed said san server using some software to a backup san drive. We wiped the orginal san drive. Used the same software to restore the backup san drive contents to the live san drive.

Months later after users had been using the directoires on the san server, they start complaining of random files being corrupt when they open them. After manualy draging and droping the old backup file form the backup san over the (same exact date and size) file in the live san, the file works for them.

The files are random that are corrupt so we need to copy all the files that exist in the backup drive into the live drive overwriting the files if they exist but ONLY if the file date is equal of older than what is there as users have been updating files and adding to the directories.

whew. and that was the short version.
wildside is offline   Reply With Quote
Old 07-17-2008   #7 (permalink)
Linux Engineer
 
wje_lf's Avatar
 
Join Date: Sep 2007
Location: Mariposa
Posts: 1,017
Quoth bigtomrodney:
Quote:
rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?
Two nits, one unimportant, one important.

The unimportant one: By "created", I assume you mean "most recently modified", correct?

The important one: By "millisecond", I hope you mean "second".

Because if it's "second", I have an idea for a solution that will take some work to whip into shape. But if you can point me to a source which shows that it's "millisecond", then I know my idea won't work, so I won't spend the time on it.

So, BTR, should I go for it? Or is it really "milliseconds"? :)
__________________
--
Bill

Old age and treachery will overcome youth and skill.
wje_lf is offline   Reply With Quote
Old 07-17-2008   #8 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Well. This is what I have so far and it "almost" works. But as my father used say; "Almost only cuts it in horseshoes and hand-grenades". So my issue is that it actually does not copy anything and just says it is newer on every single file.

Here is the script:
Code:
#!/bin/bash
find /FolderA -type f | while read f
do
    destfile=$(echo "$f" | sed 's#/FolderA#/FolderB#')
    if find "$destfile" -newer "$f"
    then
        echo "$destfile is newer than $f, skipping"
    else
        echo cp -p "$f" "$destfile"
    fi
done
I need to add that the last echo line is there for testing purposes.
wildside is offline   Reply With Quote
Old 07-17-2008   #9 (permalink)
Linux Engineer
 
wje_lf's Avatar
 
Join Date: Sep 2007
Location: Mariposa
Posts: 1,017
Allow me to tell a very short story. It involves a program that we computer operators (that should tell you how old I am) used to run on an IBM 1401 at the University of California at Irvine. It was a 36-hour run. It would print six copies of each student's class schedule on cardboard stock. It would also make nonrepeatable changes on disk, so you couldn't rerun the program if you fouled things up with the printer (easier done than said, even). You'd have to call in the programmer, who was a grouchy sort with a foul tongue and much political power. So you did whatever you had to, to make sure you didn't call him.

As luck would have it, I messed up on the printer. I don't want to tell you what I had to do to avoid calling him. I will tell you that it involved marking the bad place in the printed output, running something unauthorized at the end, and spending the intermediate hours writing a tiny machine (not assembly) language program, desk checking it several times thoroughly, and manually multi-punching it so it would be ready to run at the end.

And I learned a lesson: if the output is important, break up the task into two or more phases if possible.

So, wildside, my advice to you is this: first generate the list of files to copy, and then copy them. To do that, make these changes to your script:
Code:
#!/bin/bash
echo #!/bin/bash > copyscript.sh
find /FolderA -type f | while read f
do
    destfile=$(echo "$f" | sed 's#/FolderA#/FolderB#')
    if find "$destfile" -newer "$f"
    then
        echo echo "$destfile is newer than $f, skipping" >> copyscript.sh
    else
        echo echo cp -p "$f" "$destfile" >> copyscript.sh
        echo cp -p \"$f\" \"$destfile\" >> copyscript.sh
    fi
done
chmod 700 copyscript.sh 
Then run your first script. Then casually examine your second script, which will contain every copy command, plus echo commands for all files.

When you're satisfied, then run the second script.
__________________
--
Bill

Old age and treachery will overcome youth and skill.
wje_lf is offline   Reply With Quote
Old 07-17-2008   #10 (permalink)
Just Joined!
 
Join Date: Jul 2008
Posts: 20
Ok. So i ran the new script that exports out all the echo lines but it doesnt help me. I already knew what it was or wasn't doing. I get this:

Code:
echo /FolderB/.DS_Store is newer than /FolderA/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/.DS_Store is newer than /FolderA/AnimalToysIcons/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/Fasticon.com.url is newer than /FolderA/AnimalToysIcons/Fasticon.com.url, skipping
echo /FolderB/AnimalToysIcons/Icons/.DS_Store is newer than /FolderA/AnimalToysIcons/Icons/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/Icons/Elephant is newer than /FolderA/AnimalToysIcons/Icons/Elephant, skipping
echo /FolderB/AnimalToysIcons/Icons/Giraffe is newer than /FolderA/AnimalToysIcons/Icons/Giraffe, skipping
echo /FolderB/AnimalToysIcons/Icons/Gorilla is newer than /FolderA/AnimalToysIcons/Icons/Gorilla, skipping
echo /FolderB/AnimalToysIcons/Icons/Lion is newer than /FolderA/AnimalToysIcons/Icons/Lion, skipping
echo /FolderB/AnimalToysIcons/Icons/Zebra is newer than /FolderA/AnimalToysIcons/Icons/Zebra, skipping
echo /FolderB/AnimalToysIcons/License - Read Me.url is newer than /FolderA/AnimalToysIcons/License - Read Me.url, skipping
echo /FolderB/C_PROJECTS/conversion.cpp is newer than /FolderA/C_PROJECTS/conversion.cpp, skipping
echo /FolderB/CODES.txt is newer than /FolderA/CODES.txt, skipping
echo /FolderB/new folder/CODES.txt is newer than /FolderA/new folder/CODES.txt, skipping
I know that my script is NOT determining if the file is newer or not correctly. It thinks that even though it is the same exact mod date that it is newer so it skips it. I belive my error is here:

Code:
if find "$destfile" -newer "$f"
But i dont know what else to add to the switches to make it work.
I thought of maybe using -newerXY

I need it to copy the file as long as it exists with the same date and only skip if the dest file is newer.

Any thoughts?
wildside is offline   Reply With Quote
Reply



Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off
 

Free Magazines
Cisco News
Receive a free quarterly e-newsletter with exclusive articles on how Cisco IT uses its own products and solutions to enable the business.
subscribe
Systems Management News, the newspaper for IT systems administration and data center managers!
Each issue of Systems Management News is chock-full of news and analysis to help you understand what's happening in your field.
subscribe
The Enterprise Newsweekly
eWeek is the essential technology information source for builders of e-business.
subscribe
Oracle Magazine
Oracle Magazine contains technology strategy articles, sample code, tips, Oracle and partner news, how to articles for developers and DBAs, and more. Oracle (NASDAQ: ORCL) is the world's largest enterprise software company.
subscribe
Total Telecom
Total Telecom is "The Economist of the communications industry".
subscribe
More free magazines »



All times are GMT. The time now is 12:06 AM.




© 2000 - 2008 - All Rights Reserved - Property of  MAS Media

Content Relevant URLs by vBSEO 3.2.0