Heartbeat configuring issue

[asela115]

hi
I'm configuring linux heartbeat in my two redhat boxes
i have below few things to clarify

* can i set up heartbeat between two different notworks? (e.g primary server IP is 192.168.x.x and secondary server IP is 10.48.X.X) - which are located at different locations
* i got below error in my secondary server once primary is down (when stop heartbeat in primary server)

Code:

Oct 18 20:19:00 omnisig_smsc2 heartbeat: info: Taking over resource group 192.168.129.19
Oct 18 20:19:00 omnisig_smsc2 heartbeat: info: Acquiring resource group: omnisig_ppsmsc 192.168.129.19 httpd
Oct 18 20:19:00 omnisig_smsc2 heartbeat: info: Running /etc/ha.d/resource.d/IPaddr 192.168.129.19 start
Oct 18 20:19:00 omnisig_smsc2 heartbeat: ERROR: unable to find an interface for 192.168.129.19
Oct 18 20:19:00 omnisig_smsc2 heartbeat: warning: Return code 1 from /etc/ha.d/resource.d/IPaddr
Oct 18 20:19:00 omnisig_smsc2 heartbeat: info: Running /etc/rc.d/init.d/httpd  start
Oct 18 20:19:00 omnisig_smsc2 heartbeat: info: mach_down takeover complete.

what does that mean?

highly appreciate anybodies support on this

thanks

[asela115]

hi,

any updates on this?

[gooney0]

asela115,

From the error message provided it sounds like the secondary is attempting to take over the IP of 192.168.129.19 but your secondary server doesn't have an interface on the 192.168.129 network.

What do you want your "cluster" IP address to be? Usually you would do something like this:

Server A: 10.1.1.8
Server B: 10.1.1.16

Cluster address: 10.1.1.32

Users will access the cluster address of 10.1.1.32. Whichever server is active will control this address.

In your case you can't do this unless both servers have a network in common. Since they are in separate locations this doesn't make much sense.

If indeed you can't failover the IP address you'll need to take that out of your configuration. You could still start and stop services if you wanted to and leave the directing of traffic to another server.

-Gooney0

[asela115]

thanks gooney for your reply.
as i have understood form your reply it seems this fail over method doesn't work between different networks

and the other thin is you need to have a seperate IP as the cluster IP(rather than running the services over the management IP)

please correct me if i'm wrong

anyway as i think the best way to have a disaster recovery is to have the backup server in differnt physical location (in my case it's in a differernt IP network)

can anyone tell me a way to implement a failover machanism in such situation

thanks for your support

asela

[gooney0]

asela115,

You did understand correctly.

In most cases there will still be a single point of failure. The IP the users access will need to be location-specific.

You mentioned disaster... Do you really need failover within seconds? If not you could do something like:

Use rsync (and things like it) to duplicate data.

Use DNS to point users to the active server. goodserver = 10.1.1.16

The downside to this is DNS caching. The advantage is you probably have multiple DNS servers.

You could also use a NAT, and manually change it as needed. If your router or firewall goes dead your out of luck, but thats probably always true anyway.

Most HA solutions require the servers to share a physical location. Perhaps you'd rather have a "hot spare?"

If you went the DNS route you could write a script to automate that. It has its drawbacks but it sure is cheap.

-Gooney0

[asela115]

thanks gooney for your helping hand

actually this is a SMPP server (related to SMS services). so I believe the DNS thing will not work for me.

and this is in production environment where all the application servers get connected to it (yes I'm not doing testing over it but i have been asked to implement a fail over mechanism to it) like Facebook servers.

so this is a bit serious business and i need to find the most optimal mechanism

if you can further assist me to clear below things, it would be great

• do i need to run my service (in my case it's SMPP service) over a different IP rather than the management IP itself (below is my ifconfig output) - I'm highlighting this since it's already in live environment so changing IP configurations is difficult for me
  
  eth0 Link encap:Ethernet HWaddr 00:0E:7F:64:53:E6
  inet addr:10.48.248.238 Bcast:10.48.248.255 Mask:255.255.255.224
  inet6 addr: fe80::20e:7fff:fe64:53d6/64 Scope:Link
  UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
  RX packets:4839970 errors:0 dropped:4226 overruns:0 frame:0
  TX packets:1027020 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:2473082641 (2.3 GiB) TX bytes:314359332 (299.7 MiB)
  Interrupt:201
  
  lo Link encap:Local Loopback
  inet addr:127.0.0.1 Mask:255.0.0.0
  inet6 addr: ::1/128 Scope:Host
  UP LOOPBACK RUNNING MTU:16436 Metric:1
  RX packets:4989 errors:0 dropped:0 overruns:0 frame:0
  TX packets:4989 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:0
  RX bytes:2651529 (2.5 MiB) TX bytes:2651529 (2.5 MiB)
  

• if i really have to go with different networks with different physical locations, what are the available options (I'm not aware about rsync) ?

[gooney0]

For heartbeat yes you must use a different IP. Here is why:

If the primary server gets overloaded it will still hold it's IP address. When the secondary takes over you'll have an IP address conflict. You'll have a real mess when you reboot the primary.

Here is a practical example for fast failover:

Primary: 10.48.248.238
Secondary: 10.48.248.239
Cluster: 10.48.248.240

On a second NIC for each machine there is a heartbeat network:
Primary: 192.168.1.1
Secondary: 192.168.1.2

(If you don't use a separate NIC and private network for heartbeats you WILL be sorry.)

DNS: server.example.com = 10.48.248.240

Normally users connect to 10.48.248.240 which goes to Primary. Should the primary fail...

10.48.248.240 is released by Primary. Secondary applies 10.48.248.240 to it's NIC, and starts up services if needed.

rsync is a program / service to sync files between systems. See man rsync

You'll need to setup test machines to get all this working. There are so many subtle details to iron out in failover setups.

When I developed our Frontdoor firewall appliance with heartbeat I spent weeks getting all the failure possibilities ironed out. Write down all the possible failures and test:

Either system goes down.
Either system becomes unusable but doesn't go down.
heartbeat network fails.
Main NIC or network fails.

To help with some of these you'll need to actively monitor the primary for failures. Should it fail a test it should commit suicide and allow the secondary to take over. I used "mon" for this.

Google /etc/mon.d for some details.

-Gooney0

[gooney0]

If you need different locations:

You'll want to have a "director." Such as:

server.you.com = 10.1.1.1

The "director" will then decide which server to use. If the director dies you're hosed, so you will often have 2 of them.

Google: "Linux Virtual Server Project" or LVSP

You could also use a product like "Big IP" to do this.

Yet another method would be to rent space in a cloud such as Amazon's E3.

-Gooney0

[asela115]

thanks gooney
it saves a lot of time. I'll follow these guides.


thanks for your generosity