Find the answer to your Linux question:
Results 1 to 9 of 9
Hi there, I need to create a script that will compare the differences between two folders and then to copy only the updated and new files only to another directory. ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Oct 2010
    Posts
    4

    Talking rsync script help needed


    Hi there,

    I need to create a script that will compare the differences between two folders and then to copy only the updated and new files only to another directory. I know I need to use rsync here, I can write scripts so really it not how to create a script it is how do I accomplish the transfer of only new or changes files between two folders to a new file.


    Do I need to link these two folders first and then use the "--compare-dest" switch. Any help would be much appreciated.


    Brian
    Last edited by borgig; 10-05-2010 at 03:24 PM.

  2. #2
    Just Joined!
    Join Date
    Jun 2010
    Posts
    2
    Why not just mirror? it does compression and verification. Simple, effective. (?)

    (from man rsync)
    I mirror a directory between my "old" and "new" ftp sites with the command:

    rsync -az -e ssh --delete ~ftp/pub/samba nimbus:"~ftp/pub/tridge"

    This is launched from cron every few hours

  3. #3
    Just Joined!
    Join Date
    Dec 2007
    Posts
    16
    Is this three folders?

    I can interpret 3 different situations from what you described:
    1) differences between folder A and folder B are put in folder C
    2) folder A and folder B need to match and both are changing
    3) folder A and folder B need to match and only A is changing

    In the case of the first, you'll have to explain a bit more what you want. I do something like that when I run backups. I like to keep the changes, so I run:
    Code:
    rsync -a --link-dest="Folder that was my previous backup" "folder I'm backing up" "location I'm putting my current backup"
    For example:
    Code:
    rsync -a --link-dest=/backup/2010-10-04 /home/cong /backup/2010-10-05
    In the case of the first backup, since there's no "previous backup" just make sure you pass nothing to --link-dest=

    In the case of the second, rsync is not the thing you want. (try unison)

    In the case of the third. borgig gave a good example.

    Let me know if we can help in other ways with rsync.

  4. #4
    Just Joined!
    Join Date
    Oct 2010
    Posts
    4
    cong06,

    the 1st situation is exactly what i need to figure out mate,
    For example
    I have three folders A,B and C.
    A and B are copies of each other.
    However during the day new data will be placed in A by users.
    At the end of the day I want A to compare against B and have only the differences ie: (new files and folders or modified files) copied to C.

    So from what you have posted above my command would look like this? "rsync -a --link-dest=/B /A /C"

    I have tried this but when I run it all data from A gets written to C,

    Borgig

  5. #5
    Just Joined!
    Join Date
    Oct 2010
    Posts
    4
    evensbw,

    I only need new and updated files written to a new location, I have explained a bit better in my post to cong06 of what I am trying to do. Would a mirror just copy the difference from one folder to another? When i need the difference written to a separate folder.

    Borgig ,

  6. #6
    Just Joined!
    Join Date
    Nov 2007
    Posts
    7
    I cannot think of a solution that does this with only using rsync

    I tried to think of a solution in which you combine find and the -exec option to execute rsync.

    But those programs are not meant to "carry over" a third argument.
    You essentially have 3 arguments. And neither of thoe programs are designed to work with that.

    I think you should write a simple shell script where you might use find, a variable to compare if the object is found or not in the other directory and only if it matches do a rsync.

  7. #7
    Just Joined!
    Join Date
    Dec 2007
    Posts
    16
    Quote Originally Posted by borgig View Post
    cong06,

    the 1st situation is exactly what i need to figure out mate,
    For example
    I have three folders A,B and C.
    A and B are copies of each other.
    However during the day new data will be placed in A by users.
    At the end of the day I want A to compare against B and have only the differences ie: (new files and folders or modified files) copied to C.

    So from what you have posted above my command would look like this? "rsync -a --link-dest=/B /A /C"

    I have tried this but when I run it all data from A gets written to C,

    Borgig
    Yes. When you use --link-dest=, you'll find that all the files of A are in C. I like that for what I'm doing (it uses hard links, so I can keep the files altogether without taking up extra space)

    What I would suggest, since it seems you just want to copmare them, is to just do a "--dry-run" or -n, search for "-n, --dry-run" in your man page.
    Code:
    rsync -av --dry-run /A /B
    This code will act as if it's copying changes from A to B, but instead of preforming the changes, it will simply display them.

    Note: I've been using the '-a' switch, because I enjoy it on my Linux system. You'll want to check and see which file settings are important to you, and copy just those. Check the man page to see the different switches.

    Even if this isn't exactly what you want (ie, you want to actually move files), I'd start with this, and then what you can do is copy the files that show up into another directory.

    Does that help?

  8. #8
    Just Joined!
    Join Date
    Oct 2010
    Posts
    4
    Hi all,

    I finally got it to work,

    I am using this :

    rsync --backup --backup-dir=$(date +%Y.%m.%d) -a /home /homebackup

    What this does is replace the old files and folders with updated files and folders and transfers old files and folders to a folder titled "current date" ie "2010-10-05" within the backup directory. works like a charm. Thanks to all you guys for your support. You ideas enabled me root out the right solution.

    Cong06 big shout to you as it was your post helped me the most

    Thanks again guys
    Borgig

  9. #9
    Just Joined!
    Join Date
    Dec 2007
    Posts
    16
    Glad you got it working.

    Actually... I need to look into that "--backup" option, cause I'm doing the same thing except with more of a mess.

    I'm wondering if the backup option uses hardlinks like --link-dest? Could someone that knows more clarify?

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •