Production servers HD backup

[jordi]

Hello,

I administer several web hosting (combined with mail relays and other services) production servers under Debian GNU/Linux. I began giving these public services two years ago via three boxes: the first is a gateway which controls traffic via iptables (it's attached to a DSL modem) between a public subnet (the DMZ) and a local network which connects several workstations. In the DMZ subnet I maintain two Pentium-III era boxes, they've grown in services since I set them up. Actually, I think I should buy new ones, but, you know, I want to save money and lenghten its life.

So, they've grown in data hosted, but I've never implemented a resilent backup system. I've set up some rsync tasks sheduled via cron jobs to copy the entire UNIX file system in each of the DMZ boxes, but I'd like to be prepared before an unexpected "real" crash of some HDD, I mean, some problem that renders a disk unusable.

AFAIK, sysadmins sync entire HD backups which are capable of recovering a system via swapping the unusable unit with the backup unit. Maybe the best fashion is to implement a RAID mirroring the unit, I'm I right? So, keeping my systems as they are, I mean, capable of using 4 parallel ATA units, what would you do? Use dump, rsync or some other way to have an operational second unit with an exact copy in a bootable second drive, in order to quickly swap it if the main unit fails?

Comes to my mind to partition a second unit (so making it bootable) and backup daily via rsync only those parts of the unix file system hierarchy which are necessary to boot a system properly. What do you think about this workaround?

I'll appreciate your comments, cause I don't really know what is the best way to achieve this... Thank you!

[HROAdmin26]

Use dump, rsync or some other way to have an operational second unit with an exact copy in a bootable second drive, in order to quickly swap it if the main unit fails?

Why do *anything* manually if you don't have to? This is where the md driver can be used to create a RAID1 (mirror) set. If a drive fails, you get notified, but the server continues running. For a "production" server this is highly desirable.

BUT, RAID is *not* a backup. If data is deleted from the mount point, it's removed from both drives. That's not a backup.

Use whatever solution makes sense for your backups. If you can zip up your data and rsync it to another drive/server (keeping multiple revisions, ideally) then continue doing so. There are lots of commercial solutions as well as Bacula, BackupPC, etc.

[jordi]

Why do *anything* manually if you don't have to?

Sorry, but i don't know how RAID works...

Correct me if I'm wrong: RAID1 is implemented via 2 HDD, and there's no need of special hardware (md driver could be used under Debian)?

[Irithori]

Discimages are fine, it is simple to restore a system with them exactly as it was at the time of the creation of that image.

But there are two problems:
- creating them while the server is doing duty is difficult at least
- if the raid/disclayout of the new server is significantly different, restoring might be tricky

What I do:
- I consider a re-install simple.
Ok, I do have a pxe based installation system. Kickstart and cfengine will do all the basic work (partitioning, network, ntp, monitoring, backup client, etc)
But even a manual install from CD is not that much work.
- My configs of apache, mysql, postgres, imap, tomcat, etc are in a subversion repository and spread out by cfengine.
So I just have to include a new server in the right groups.
- There is a always a daily file backup of all important servers. This way a new server can get whatever data it needs.

For a start:
- A raid is no backup. I consider a raid
a) a way of buying time and avoiding unnecessary work. If a disc fails -and they all will at some point- it is simple to get a replacement.
b) a way of gaining IO performance.
- You can filebackup the whole disc and just leave out the volatile directorys ( /proc, /dev, DB datadirs, etc) + ensure that the DBs dump their data regularly.

After a re-install
- re-install all needed server packages apache, mysql, php, whatever
- recover data and /etc directory from backup
- cp the recovered config files in place (this might need some fiddling if done manually)
- move the recovered data where the daemons can find it.


My backupsystem of choice is Bacula, the Open Source, Enterprise ready, Network Backup Tool for Linux, Unix, and Windows , and I find it quite reliable and versatile

[jordi]

Clearly, RAID is no backup So, apart of the necessary incremental copy, what utility/resource should I use under debian to implement mirroring (RAID1)? Must the HDD's be identical (capacity, etc)?

[Irithori]

The HDs must not have the same capacity, but a smaller one would determine the capacity.
Usually, the same model of drive is used.
A software raid is easiest to setup during a fresh install.

Converting a existing server from a single disc to raid1 is possible but requires some tricks and several data copies.

I would advise doing a fresh install with two new drives and then copy over config + data from the original disc

[HROAdmin26]

Wikipedia - RAID

Google: how to implement raid md debian

[jordi]

Definitely, thank you .