backup bash script advice

[thomas130]

Hi guys I new to linux and bash and created this bash backup script. I was wondering can look at it and see if I doing anything wrong or if there is a better way of doing it etc.

Thanks

#---------------------------------------------------------------------------------
#Backup Paths
array[0]=/home/ops/Desktop/dir1
array[1]=/home/ops/Desktop/dir2
array[2]=/etc/dir3

TEMP="/home/ops/Desktop/temp"
BACKUPLOCATION="/Tom/Backup"
MD5="/home/ops/Desktop/check/check1.md5"
#--------------------------------------------------------------------------------

#E-MAIL
EMAIL="test@local.com"
EFTAR="Tar has failed"
SUCCESS="Backup has completed"
EBACK="Backup has failed"

#variable for dates
Today=`date +%d-%m-%Y`
TodayTar=/mnt/ntserver/Tom/Backup-$Today.tar
#---------------------------------------------------------------------------------


#------------------------------------------------------------------------------------------
#Make sure volume is mounted at start of script compress folder and copy to pulse-nt-live
mount -t cifs //pulse-nt-live/computers -o username=user,password=user /mnt/ntserver
cd $TEMP
tar czvf backup-"$(date +%d-%b-%y)".tar ${array[@]}
if [ $? -eq 0 ];
then
cp /$TEMP/backup-"$(date +%d-%b-%y)".tar /mnt/ntserver/$BACKUPLOCATION
else
mail -s "$EFTAR" "$EMAIL" </dev/null
exit
fi
#-------------------------------------------------------------------------------------------

# check copy for corruption
md5sum backup-"$(date +%d-%b-%y)".tar > $MD5
cd /mnt/ntserver/$BACKUPLOCATION
md5sum -c $MD5
if [ $? -eq 0 ];
then
find $BACKUPLOCATION -type f -name "*.tar" -mtime +8 -delete && find $TEMP -type f -name "*.tar" -mtime +8 -delete && mail -s "$SUCCESS" "$EMAIL" </dev/null
else
mail -s "$EBACK" "$EMAIL" </dev/null
fi
#----------------------------------------------------------------------------------------------

[ramin.honary]

It looks pretty good so far, just a few pointers I had.
First:

#Backup Paths
array[0]=/home/ops/Desktop/dir1
array[1]=/home/ops/Desktop/dir2
array[2]=/etc/dir3

I find that the list of files I want to backup (and not backup) is always changing, so I try to keep the list of files I want to backup separate from the script. Lets say you have two files: "backup-include.list and backup-exclude.list. You can use the tar command like this:

tar xzvf backup.tgz $(cat backup-include.list) -X backup-exclude.list

The "-X" paramter to GNU tar will read a file with filenames and directories listed in the file, line-by-line. This file may include globs (wildcards characters) as well. The "$(cat include.list)" uses the "cat" utility to expand the list of files into the command-line parameters to tar.

Second:

#variable for dates
Today=`date +%d-%m-%Y`
TodayTar=/mnt/ntserver/Tom/Backup-$Today.tar

It is better to name your files YEAR-MONTH-DAY(-TIME) so when you have lots of files, sorting the list by name using "ls" or "sort" will also be sorting the list by date. This is especially handy if, after transferring your backup files from one disk to another, or over a network where the timestamps are not kept consistent, you happen to loose the date timestamps on all the files and "ls -t" doesn't list items by date anymore.

Third:

cd $TEMP
tar czvf backup-"$(date +%d-%b-%y)".tar ${array[@]}
if [ $? -eq 0 ];
then
cp /$TEMP/backup-"$(date +%d-%b-%y)".tar /mnt/ntserver/$BACKUPLOCATION

You use the `date` command on two different lines in this snippet (and three different places in your whole script). Suppose your script executes at 23:59:59.999 (one milisecond before midnight)? The first filename created by `date` might turn out to be different than the second filename created by `date`. It is better to store the `date` string in a variable ONCE and use that variable throughout the script.

Fourth:

cd $TEMP
tar czvf backup-"$(date +%d-%b-%y)".tar ${array[@]}
if [ $? -eq 0 ];
then cp /$TEMP/backup-"$(date +%d-%b-%y)".tar /mnt/ntserver/$BACKUPLOCATION
fi

When creating backups, it is really not a good idea to place it in a temporary location first. It takes time, and increases the chance of failure. Make the backup file once in the place where you want it to go.

Finally, you specify the parameters "czvf" to the "tar" command: Create, gZip, Verbose, File -- since you "gzip" the archive, you should use thr ".tgz" or ".tar.gz" file extension.

Today=$(date '+&Y-%m-%d)
BackFile="backup-${Today}.tgz"
MD5="/home/ops/Desktop/check/check1.md5"
if [ -e "backup-exclude.list" ]
then tar czvf $BACKUPLOCATION/$BackFile $(cat "backup-include.list") -X backup-exclude.list
else tar czvf $BACKUPLOCATION/$BackFile $(cat "backup-include.list")
fi
md5sum $BackFile > $MD5

So that is how I would write the "tar" and "md5sum" commands in your script.

Let us know if you have anymore questions!

[alf55]

Usages of:

${array[@]}

should be

"${array[@]}"

[thomas130]

It looks pretty good so far, just a few pointers I had.
First:I find that the list of files I want to backup (and not backup) is always changing, so I try to keep the list of files I want to backup separate from the script. Lets say you have two files: "backup-include.list and backup-exclude.list. You can use the tar command like this:

tar xzvf backup.tgz $(cat backup-include.list) -X backup-exclude.list

The "-X" paramter to GNU tar will read a file with filenames and directories listed in the file, line-by-line. This file may include globs (wildcards characters) as well. The "$(cat include.list)" uses the "cat" utility to expand the list of files into the command-line parameters to tar.

Second:It is better to name your files YEAR-MONTH-DAY(-TIME) so when you have lots of files, sorting the list by name using "ls" or "sort" will also be sorting the list by date. This is especially handy if, after transferring your backup files from one disk to another, or over a network where the timestamps are not kept consistent, you happen to loose the date timestamps on all the files and "ls -t" doesn't list items by date anymore.

Third:You use the `date` command on two different lines in this snippet (and three different places in your whole script). Suppose your script executes at 23:59:59.999 (one milisecond before midnight)? The first filename created by `date` might turn out to be different than the second filename created by `date`. It is better to store the `date` string in a variable ONCE and use that variable throughout the script.

Fourth:
When creating backups, it is really not a good idea to place it in a temporary location first. It takes time, and increases the chance of failure. Make the backup file once in the place where you want it to go.

Finally, you specify the parameters "czvf" to the "tar" command: Create, gZip, Verbose, File -- since you "gzip" the archive, you should use thr ".tgz" or ".tar.gz" file extension.

Today=$(date '+&Y-%m-%d)
BackFile="backup-${Today}.tgz"
MD5="/home/ops/Desktop/check/check1.md5"
if [ -e "backup-exclude.list" ]
then tar czvf $BACKUPLOCATION/$BackFile $(cat "backup-include.list") -X backup-exclude.list
else tar czvf $BACKUPLOCATION/$BackFile $(cat "backup-include.list")
fi
md5sum $BackFile > $MD5

So that is how I would write the "tar" and "md5sum" commands in your script.

Let us know if you have anymore questions!

That looks great thanks for your advice it's been a great help

[Kloschüssel]

Couldn't rsync do much of the work? Or something like rdiff-backup? Or even a repository (hg, git, ..) that is automatically pushed to a remote location (i.e. a backup server within a lan)?

[ramin.honary]

Couldn't rsync do much of the work? Or something like rdiff-backup? Or even a repository (hg, git, ..) that is automatically pushed to a remote location (i.e. a backup server within a lan)?

Well, this is the Programming/Scripting thread, I assumed Thomas130 was more interested in creating their own script.

Actually, though "rsync", "git", and "svn" are incredibly handy and easy to use, I find that "tar" is just good enough for most of my backup needs. I have everything on my laptop, and I don't care about regular backups (I don't even know when the next time I will be able to run a backup is). I just want something that dumps a bunch of files into an archive labeled by date into the same directory whenever I plug in my USB hard disk backup drive. To revert to a previous version, just untar the desired tar file to the desired location, no fussing with git or svn commands.

Also, I don't know about "git", but I know "svn" creates all kinds of meta-data in every subdirectory under revision control, which can be a real pain to deal with at times.

Sometimes a sleek, network-ready solution is overkill.

[Kloschüssel]

Sometimes a sleek, network-ready solution is overkill.

I agree. But most often it is a bad idea to reinvent the weel and it's worth investigating new/other technologies. Most often I don't trust my own code as I may haven't thought of all problems that could happen. Therefore I prefer to take things that thousands of other people are using and if they had found a problem, it would have been solved long ago. Plus, using a repository doesn't necessarily require it to reside on a remote server. Using hg, the "remote repository" could be your usb drive.

I was wondering can look at it and see if I doing anything wrong or if there is a better way of doing it etc.

Therefore I wanted to point the poster to some ideas that I personally believe they are worth the time looking at.

[ramin.honary]

@Klischussel
Cool, good point.

[RobKendrick]

ramin.honary's initial reply to this was very helpful for me as well, as I didn't think of an include/exclude list in my own scripting; I instead separated my files into sections of important info to be backed up and non-critical data I didn't care to lose or couldn't easily sync (like files in use on Windows machines).

Kloschüssel's info has now got me looking into the man pages for hg and rdiff-backup, as the only other backup option I was initially aware of was rsync, and I was opting for tar and scp for my own backups.

Thanks for the great info, guys!
Take care,
Rob