Strannge issue with nightly rsync

[bomahony]

I hope this is the right forum area.

Im getting a strange error nightly on one of my servers using rsync. The two servers are in the same subnet (255.255.248.0), but in different locations (Ireland & BLR). However we have an MPLS between the sites, so connectivity isnt an issue.

There is about 11 rsync's initiated at the far site (BLR) copying data to dublin at around 00:30 GMT. There isnt a whole lot of data as it is using the update option.

My issue is the first rsync every night fails:

From: Cron Daemon []
Sent: Friday, July 09, 2010 12:31 AM
To:
Subject: Cron <> /root/testsync

spawn /usr/bin/rsync -av -e ssh --delete --bwlimit=0 buzz:/storage/test /storage/test
ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(165)
send: spawn id exp3 not open
while executing
"send "password\r""
(file "" line 6)

Its ONLY the first rsync. To circumvent this, at the moment, i have started the first one a minute before hand, and again at its regular time. It runs fine then. Some of the scripts then run again about 9am GMT and they work fine (without the first failing).

Does anyone have any ideas?

[zombykillah]

Your problem isn't with rsync.
It's with ssh ... it seems like a failure in the authentication.

Why that only happens with the first atempt ... I don't know.
You can try to add the hostname and IP of the rsync starter in the hosts file from the server it's connecting to.

[bomahony]

Ive added the host name to both servers and the issue still occurs (I have to wait for the automated test nightly as it works fine when manually run).

Im now wondering if its a possibility with the MPLS line timing out. That would be weird though.

[bomahony]

through in a ping to the remote server a minute before the first rsync

Still the same problem.

Anyone have any other thoughts?