Help. noob needs help with file copy!

[wildside]

So I have a very simple question that has come to be a huge headache.

I need to copy files from FolderA into FolderB replacing all files that already exist in FolderB UNLESS the date of the file in FolderB is newer than the date from the same file in FolderA. In the case that file is newer in the destination, I need to skip copy of that file and move on.

I have tried so many different things and nothing has been able to do what I need.

I have tried a simple cp -i but i need to filter by date and choose "don't overwrite" if file being copied is older, but owverwite is file date is same.

there is a long story to go along with this as well.

Thanks in advance

[Ben Cotton]

Sounds like what you need is the rsync command:

rsync -a /path/to/source /path/to/destination

See the rsync man page for more options. rsync is a very powerful utility.

[wildside]

Thanks. that is one I have not man'd yet. I will check it out.

[wildside]

Thank you for the rsync pointer.

So rsync looks like what i need. I ran some "dry-runs" with it and also some test runs.
I dont think -a (append) is what i want since that would write data to the end of the file and not replace the whole file.

using this convention:
$ rsync -vru --progress FolderA/ FolderB/

It only copies files from FolderA that are newer than files in FolderB.
It skips files that are the same size or date.
This is an expect from the rsync man page:

-u, --update
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file's, it will be updated if the sizes are different.)


So what i need to do is force rsync to use update to copy a file if it is equal or newer. NOT if it is ONLY newer.


Any ideas here?

[bigtomrodney]

Not to avoid answering your question, but surely if the date is equal surely it doesn't need to update?

rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?

[wildside]

Not to avoid answering your question, but surely if the date is equal surely it doesn't need to update?

rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?

Not so. Now for the long story.

Ok a really short version of the long story.

We have a San Server with 43TB of user data. We backed said san server using some software to a backup san drive. We wiped the orginal san drive. Used the same software to restore the backup san drive contents to the live san drive.

Months later after users had been using the directoires on the san server, they start complaining of random files being corrupt when they open them. After manualy draging and droping the old backup file form the backup san over the (same exact date and size) file in the live san, the file works for them.

The files are random that are corrupt so we need to copy all the files that exist in the backup drive into the live drive overwriting the files if they exist but ONLY if the file date is equal of older than what is there as users have been updating files and adding to the directories.

whew. and that was the short version.

[wje_lf]

Quoth bigtomrodney:

rsync doesn't work on days, it goes right down to the millisecond, so I'm guessing the likelihood of two files with the same name, created at the same millisecond being different is quite slim?

Two nits, one unimportant, one important.

The unimportant one: By "created", I assume you mean "most recently modified", correct?

The important one: By "millisecond", I hope you mean "second".

Because if it's "second", I have an idea for a solution that will take some work to whip into shape. But if you can point me to a source which shows that it's "millisecond", then I know my idea won't work, so I won't spend the time on it.

So, BTR, should I go for it? Or is it really "milliseconds"? :)

[wildside]

Well. This is what I have so far and it "almost" works. But as my father used say; "Almost only cuts it in horseshoes and hand-grenades". So my issue is that it actually does not copy anything and just says it is newer on every single file.

Here is the script:

#!/bin/bash
find /FolderA -type f | while read f
do
    destfile=$(echo "$f" | sed 's#/FolderA#/FolderB#')
    if find "$destfile" -newer "$f"
    then
        echo "$destfile is newer than $f, skipping"
    else
        echo cp -p "$f" "$destfile"
    fi
done

I need to add that the last echo line is there for testing purposes.

[wje_lf]

Allow me to tell a very short story. It involves a program that we computer operators (that should tell you how old I am) used to run on an IBM 1401 at the University of California at Irvine. It was a 36-hour run. It would print six copies of each student's class schedule on cardboard stock. It would also make nonrepeatable changes on disk, so you couldn't rerun the program if you fouled things up with the printer (easier done than said, even). You'd have to call in the programmer, who was a grouchy sort with a foul tongue and much political power. So you did whatever you had to, to make sure you didn't call him.

As luck would have it, I messed up on the printer. I don't want to tell you what I had to do to avoid calling him. I will tell you that it involved marking the bad place in the printed output, running something unauthorized at the end, and spending the intermediate hours writing a tiny machine (not assembly) language program, desk checking it several times thoroughly, and manually multi-punching it so it would be ready to run at the end.

And I learned a lesson: if the output is important, break up the task into two or more phases if possible.

So, wildside, my advice to you is this: first generate the list of files to copy, and then copy them. To do that, make these changes to your script:

#!/bin/bash
echo #!/bin/bash > copyscript.sh
find /FolderA -type f | while read f
do
    destfile=$(echo "$f" | sed 's#/FolderA#/FolderB#')
    if find "$destfile" -newer "$f"
    then
        echo echo "$destfile is newer than $f, skipping" >> copyscript.sh
    else
        echo echo cp -p "$f" "$destfile" >> copyscript.sh
        echo cp -p \"$f\" \"$destfile\" >> copyscript.sh
    fi
done
chmod 700 copyscript.sh

Then run your first script. Then casually examine your second script, which will contain every copy command, plus echo commands for all files.

When you're satisfied, then run the second script.

[wildside]

Ok. So i ran the new script that exports out all the echo lines but it doesnt help me. I already knew what it was or wasn't doing. I get this:

echo /FolderB/.DS_Store is newer than /FolderA/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/.DS_Store is newer than /FolderA/AnimalToysIcons/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/Fasticon.com.url is newer than /FolderA/AnimalToysIcons/Fasticon.com.url, skipping
echo /FolderB/AnimalToysIcons/Icons/.DS_Store is newer than /FolderA/AnimalToysIcons/Icons/.DS_Store, skipping
echo /FolderB/AnimalToysIcons/Icons/Elephant is newer than /FolderA/AnimalToysIcons/Icons/Elephant, skipping
echo /FolderB/AnimalToysIcons/Icons/Giraffe is newer than /FolderA/AnimalToysIcons/Icons/Giraffe, skipping
echo /FolderB/AnimalToysIcons/Icons/Gorilla is newer than /FolderA/AnimalToysIcons/Icons/Gorilla, skipping
echo /FolderB/AnimalToysIcons/Icons/Lion is newer than /FolderA/AnimalToysIcons/Icons/Lion, skipping
echo /FolderB/AnimalToysIcons/Icons/Zebra is newer than /FolderA/AnimalToysIcons/Icons/Zebra, skipping
echo /FolderB/AnimalToysIcons/License - Read Me.url is newer than /FolderA/AnimalToysIcons/License - Read Me.url, skipping
echo /FolderB/C_PROJECTS/conversion.cpp is newer than /FolderA/C_PROJECTS/conversion.cpp, skipping
echo /FolderB/CODES.txt is newer than /FolderA/CODES.txt, skipping
echo /FolderB/new folder/CODES.txt is newer than /FolderA/new folder/CODES.txt, skipping

I know that my script is NOT determining if the file is newer or not correctly. It thinks that even though it is the same exact mod date that it is newer so it skips it. I belive my error is here:

if find "$destfile" -newer "$f"

But i dont know what else to add to the switches to make it work.
I thought of maybe using -newerXY

I need it to copy the file as long as it exists with the same date and only skip if the dest file is newer.

Any thoughts?