Script to perform Rsync and wipe

[zippy4251]

Hello,

Let me start by saying that I have very little experience with scripting. I am wondering if anybody here know of or could assist in the writing of a script that would perform a rsync to a remote server, double check the integrity of those files, and then use dc3dd to wipe the source drive. So to give an example, drive sdb would have x number of files with the extention of .jpg. The rsync would copy those files to a remote server and those remote files would be checked (possibly through the use of a hash?) for integrity and ensure that all files were copied. Then, dc3dd would be used to wipe sdb. That command would look like dc3dd wipe=/dev/sdb.

I mention rsync because it's proven that it works with cifs file system that we use on the server and it's quicker than doing a simple copy. If there is a better way of copying the files without sacrificing too much speed, I'm open to suggestions. I should also mention that the files range from a few megabytes in size to several terabytes.

If anyone could assist with this, it would be greatly appreciated.

[amlife]

example

dir A has all of your files and B is the destination, you may use --remove-sent-files option with rsync to delete files from sender after copying files over to B

rsync -ac --progress --remove-sent-files A/ B/

-a to preserve permission and -c to do md5 checksum

keep in mind -c check will slow things down! you may consider to use --whole-file option when copying large files, as it will copy files whole (without rsync algorithm) and it will speed up copying process.

cheers

[zippy4251]

example

dir A has all of your files and B is the destination, you may use --remove-sent-files option with rsync to delete files from sender after copying files over to B

That's a good point. I could use that for some things, but I'll need to do a complete wipe and zero write after each copy usually. Unless I'm mistaken, the option you mention will only do a simple delete of the files right?

EDIT: Grammar purposes

[amlife]

Yes it will only delete! why would you need to wipe! and zero?

this process might become a CPU intensive and don't get me wrong it "doesn't make any scene" to wipe/format drive after each sync!

If you are concerned about security, I recommend to encrypt your images or the file system! there are many tools to do that!

anyhow

You may use any scripting language (bash/perl/python)

however lets do this in bash (assuming you have downloaded and compiled dc3dd tool!

Don't run the script without taking backup! or at least run it on a test machine!

#!/bin/bash

#lets get the sync opration out of the way! 


rsync -ac --delete --remove-sent-files /root/images/ /opt/images/

if [ $? == 0 ]

then

        echo "wiping drive"; dc3dd wipe=DRIVE sizeprobe=on progresscount=10 bs=4M

else
        echo "there seems to be an issue with rsync"

fi

[zippy4251]

Yes it will only delete! why would you need to wipe! and zero?

You're right, this is an unusual situation and most people wouldn't do it this way. It's a specific need for me and that's why I asked for help here. I couldn't find anything like it out on the Internet and since I'm not a programmer, I had no clue how to get started on it.

If you are concerned about security, I recommend to encrypt your images or the file system! there are many tools to do that!

So do I! Many Linux distros have home folder encryption at minimum, and TrueCrypt is a great alternative.


Thank you for posting that code! I'll do some testing with it to see what happens. I wanted to throw a few thoughts of mine out there to see if they make any sense. I'm thinking that a few commands with md5sum would cover checking for file integrity. So maybe something like this?

md5sum *.*
rsync -ac /root/images /opt/images/
md5sum /opt/images/*.*

Would there be a way to include in that script a way to check the two different md5 hashes and make sure they match? I would do that before any deleting happens, so I'm thinking I wouldn't use the remove-sent-files option in the rsync. What does everyone think?

[amlife]

well, with rsync there is -c options which will automatically do md5sum check! you may use -n option to see what files are going to be copied to the other side before actual transfer process starts this is called (dry run).

you may also create a log file for all files which got transferred using --log-file=FILE file options which is associated with rsync, well I wouldn't worry too much about the integrity of the files, I have used rsync to copy hundreds of files on mission critical systems and never had an issue with that! as long as rsync operation completes without interruption everything should be just fine.

good luck!

[zippy4251]

Thanks for the help all! I've tailored the code provided to what I would need it to do.

#!/bin/bash

# This gets information from the user where the files are located and where they need to be

echo "Enter the directory to transfer files from"
read from

echo "Enter the directory to place those files into"
read into

# This gets which drive needs to be wiped after the rsync is complete
echo "Which drive needs to be wiped?"
read wipe

# This grabs a md5 hash from the source files
md5sum $from/*.txt

# This performs the rsync process. 
rsync --verbose --progress --stats --times $from/*.txt $into

# This grabs a md5 hash from the files that have been copied
md5sum $into/*.txt

# This is where the check between the md5 files should be made
if [ $? == 0 ]

then
# This will wipe the drive if the check passes. It is currently commented out.
        echo "wiping drive"
#	dc3dd wipe=$wipe

else
        echo "there seems to be an issue"

fi

This seems to work, minus the md5sum check of course. I'm not sure how something like that would be written. It would need to check the first md5sum (the source files) against the second (the target files). If it matches, the wipe continues. If not, no wipe will be performed and it will produce that error message. Anybody know how to code something like that?

[zippy4251]

Just had a random thought about the MD5 check that I need. If I could place that MD5 into a variable, I could do a simple statement seeing if the two variables match. It'd be something like if firstmd5 = secondmd5, then perform the wipe. Is it possible to do that? If so, how would I take the md5 and put it into a variable?