Heartbeat failure recovery system help

[anarchist18]

I`m trying to implement a failure recovery system for our network firewalls. In theory the idea is if one fails the other one should be able of taking over and when the master firewall comes back up online, the slave should yield its place.

I`m hoping heartbeat is the answer.

Now in practice I`m experimenting with two test systems each running FC 14 minimal install with 2 NIC`s before implementing this on the production network.

I`m trying to setup heartbeat on 2 test machines running FC 14 minimal install with 2 NIC`s

The second host should be ready to take over if host 1 is down and take it`s ip address

My test machines have the following network config

eth0 - eth0 cross cable between them with internal ip`s 10.10.0.1/24 and 10.10.0.2/24

eth2 is connected to the router 192.168.200.x

I`ve been reading fwbuilder.org/4.0/docs/users_guide/heartbeat_cluster.html however it`s not exacly what i want.

My config files:

/etc/ha.d/authkeys
auth 2
2 sha1 test


/etc/ha.d/ha.cf
logfile /var/log/ha-log
logfacility local0
keepalive 2
deadtime 30
initdead 120
udpport 694
bcast eth0
auto_failback on
node TEST1
node TEST2

haresources
TEST1 IPaddr::10.10.0.3

So this creates a virtual NIC with the address of 10.10.0.3, but i want TEST2 to take over TEST1`s ip.

How can i acheive that? Or can anyone suggest a better option than heartbeat for what i`m trying to do?

[vasile002]

you need to:
1. configure eth2 to not start on boot
2. change haresorces to:
TEST1 IPaddr2::192.168.200.x/netmask/eth2

replace netmask with the netmask of the ip

this will make heartbeat start the eth2 ip and switch it between the 2 servers

[nplusplus]

I know this is a Linux forum, but I would suggest pfSense Open Source Firewall Distribution - Home if you want something with a pretty UI and tons of configurable options, including stateful failover. You won't get stateful failover with heartbeat. If you just want stateful filtering without all of the bells and whistles, you could just try to setup PF with state synch and failover on any BSD distro (openbsd, freebsd, etc.). If you are set on Linux, you might check this out: The conntrack-tools user manual.

N

Edit: Oh, yeah, a bsd box will run much leaner than a Linux box handling the same amount of traffic (at least when you're talking FC14).

[anarchist18]

First of all I want to thank everybody for replying.
I tryed what vasile002 suggested, however it doesn`t seem to work in taking TEST1`s ip when I turn it off.
I`ll look into conntrackd but it seems pretty complicated.

All I want to do is to have the TEST2 machine take over TEST1`s ip address when it becomes unavailable and release it when TEST1 comes back online. Can anyone suggest a simple solution for this? Maybe a step by step for a newbie
It must run on FC14 because this is the OS that we use.

[anarchist18]

Seems I got it working with heartbeat finally.

cat /etc/ha.d/haresources
TEST1 IPaddr2::192.168.231.130/24 - this is TEST1`s ip that i want TEST2 to replace, must be the same on all machines.

cat /etc/ha.d/ha.cf
logfacility local0
keepalive 2
deadtime 30
initdead 120
udpport 694
bcast eth2 - interface used for heartbeats from the current host
auto_failback on
node TEST1
node TEST2

cat /etc/ha.d/authkeys
auth 2
2 sha1 test

chmod 600 /etc/ha.d/authkeys

All network interfaces start on boot and get their ip via dhcp except for TEST1 that I manually configured as 192.168.231.130.

Each host has 2 NIC`s: 1 connected to the router and the other crossed between them.

Now the ip switch works however there is no real indication just the hostname, you can see you`re on host2 but ifconfig shows the same things.

Now I want to take this a step further and try to execute a script that will copy TEST1`s settings over to TEST2 (routes, iptables). Can anyone help?

[vasile002]

for iptables you need to sync only the /etc/sysconfig/iptables file assuming that the "service iptables save" command is ran on one server whenever there is a change. you can add the iptables start command on the passive server to heartbeat

For the routes if those modify often you can run the modifications at the same time on both servers, on one via ssh

[anarchist18]

you can add the iptables start command on the passive server to heartbeat
For the routes if those modify often you can run the modifications at the same time on both servers, on one via ssh

How can I do that, can you explain please? Do I have to put "service iptables start" in ha.cf?

[vasile002]

This is how it's added:
TEST1 iptables IPaddr2::192.168.231.130/24

[anarchist18]

Ok I added exactly as you said but unfortunetly it doesn`t synchronize.
Made some changes in iptables on TEST1, they appear in /etc/sysconfig/iptables but not on TEST2.

I`m attaching the ha-log
Nov 02 13:18:39 TEST2 heartbeat: [1708]: info: Heartbeat shutdown in progress. (170
Nov 02 13:18:39 TEST2 heartbeat: [3345]: info: Giving up all HA resources.
ResourceManager[3358]: 2011/11/02_13:18:40 info: Releasing resource group: test1 IPaddr2::192.168.231.130/24 iptables
ResourceManager[3358]: 2011/11/02_13:18:40 info: Running /etc/init.d/iptables stop
ResourceManager[3358]: 2011/11/02_13:18:40 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.231.130/24 stop
IPaddr2[3473]: 2011/11/02_13:18:40 INFO: IP status = ok, IP_CIP=
IPaddr2[3450]: 2011/11/02_13:18:40 INFO: Success
ResourceManager[3519]: 2011/11/02_13:18:40 info: Releasing resource group: test1 IPaddr2::192.168.232.135/24
ResourceManager[3519]: 2011/11/02_13:18:40 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.232.135/24 stop
IPaddr2[3579]: 2011/11/02_13:18:40 INFO: IP status = ok, IP_CIP=
IPaddr2[3556]: 2011/11/02_13:18:40 INFO: Success
Nov 02 13:18:40 TEST2 heartbeat: [3345]: info: All HA resources relinquished.
Nov 02 13:18:40 TEST2 heartbeat: [1708]: info: Link test2:eth2 up.
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: killing HBFIFO process 1722 with signal 15
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: killing HBWRITE process 1723 with signal 15
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: killing HBREAD process 1724 with signal 15
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: Core process 1723 exited. 3 remaining
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: Core process 1722 exited. 2 remaining
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: Core process 1724 exited. 1 remaining
Nov 02 13:18:42 TEST2 heartbeat: [1708]: info: test2 Heartbeat shutdown complete.
Nov 02 13:21:13 TEST2 heartbeat: [1832]: info: Version 2 support: false
Nov 02 13:21:13 TEST2 heartbeat: [1832]: WARN: Logging daemon is disabled --enabling logging daemon is recommended
Nov 02 13:21:13 TEST2 heartbeat: [1832]: info: **************************
Nov 02 13:21:13 TEST2 heartbeat: [1832]: info: Configuration validated. Starting heartbeat 3.0.0
Nov 02 13:21:13 TEST2 heartbeat: [1843]: info: heartbeat: version 3.0.0
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: Heartbeat generation: 1320051021
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth2
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: glib: UDP Broadcast heartbeat closed on port 694 interface eth2 - Status: 1
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: G_main_add_TriggerHandler: Added signal manual handler
Nov 02 13:21:14 TEST2 heartbeat: [1843]: info: G_main_add_SignalHandler: Added signal handler for signal 17
Nov 02 13:21:15 TEST2 heartbeat: [1843]: info: Local status now set to: 'up'
Nov 02 13:23:15 TEST2 heartbeat: [1843]: WARN: node test1: is dead
Nov 02 13:23:15 TEST2 heartbeat: [1843]: info: Comm_now_up(): updating status to active
Nov 02 13:23:15 TEST2 heartbeat: [1843]: info: Local status now set to: 'active'
Nov 02 13:23:15 TEST2 heartbeat: [1843]: WARN: No STONITH device configured.
Nov 02 13:23:15 TEST2 heartbeat: [1843]: WARN: Shared disks are not protected.
Nov 02 13:23:15 TEST2 heartbeat: [1843]: info: Resources being acquired from test1.
harc[2609]: 2011/11/02_13:23:15 info: Running /etc/ha.d//rc.d/status status
Nov 02 13:23:15 TEST2 heartbeat: [2610]: info: No local resources [/usr/share/heartbeat/ResourceManager listkeys test2] to acquire.
mach_down[2639]: 2011/11/02_13:23:15 info: Taking over resource group IPaddr2::192.168.231.130/24
ResourceManager[2666]: 2011/11/02_13:23:15 info: Acquiring resource group: test1 IPaddr2::192.168.231.130/24 iptables
IPaddr2[2694]: 2011/11/02_13:23:16 INFO: Resource is stopped
ResourceManager[2666]: 2011/11/02_13:23:16 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.231.130/24 start
IPaddr2[2795]: 2011/11/02_13:23:16 INFO: ip -f inet addr add 192.168.231.130/24 brd 192.168.231.255 dev eth3
IPaddr2[2795]: 2011/11/02_13:23:16 INFO: ip link set eth3 up
IPaddr2[2795]: 2011/11/02_13:23:17 INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp-192.168.231.130 eth3 192.168.231.130 auto not_used not_used
IPaddr2[2772]: 2011/11/02_13:23:17 INFO: Success
ResourceManager[2666]: 2011/11/02_13:23:17 info: Running /etc/init.d/iptables start
mach_down[2639]: 2011/11/02_13:23:17 info: Taking over resource group IPaddr2::192.168.232.135/24
ResourceManager[2910]: 2011/11/02_13:23:17 info: Acquiring resource group: test1 IPaddr2::192.168.232.135/24
IPaddr2[2938]: 2011/11/02_13:23:17 INFO: Resource is stopped
ResourceManager[2910]: 2011/11/02_13:23:17 info: Running /etc/ha.d/resource.d/IPaddr2 192.168.232.135/24 start
IPaddr2[3039]: 2011/11/02_13:23:17 INFO: ip -f inet addr add 192.168.232.135/24 brd 192.168.232.255 dev eth3
IPaddr2[3039]: 2011/11/02_13:23:17 INFO: ip link set eth3 up
IPaddr2[3039]: 2011/11/02_13:23:18 INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/heartbeat/rsctmp/send_arp-192.168.232.135 eth3 192.168.232.135 auto not_used not_used
IPaddr2[3016]: 2011/11/02_13:23:18 INFO: Success
mach_down[2639]: 2011/11/02_13:23:18 info: /usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
mach_down[2639]: 2011/11/02_13:23:18 info: mach_down takeover complete for node test1.
Nov 02 13:23:18 TEST2 heartbeat: [1843]: info: mach_down takeover complete.
Nov 02 13:23:18 TEST2 heartbeat: [1843]: info: Initial resource acquisition complete (mach_down)
Nov 02 13:23:26 TEST2 heartbeat: [1843]: info: Local Resource acquisition completed. (none)
Nov 02 13:23:26 TEST2 heartbeat: [1843]: info: local resource transition completed.

[vasile002]

heartbeat can't take care of the syncing of the rules. You need to use some cron job to do that. heartbeat can only stop/start/restart iptables