I'm running ntpd on my servers to keep their times synced with my 2 ntp servers (both windows 2003 server domain controllers).

The problem is that the ntpd is not keeping the time accurate and the clocks are drifting. I'm sure that when ntp is working properly the time is supposed to be second accurate, but it's not happening here.

The clocks are drifting 15-180 seconds away from the time of the master ntp servers within a few days. I then have to ntpdate to adjust the clocks back to the right time manually which is silly (in fact I have to stop ntpd, then ntpdate, then start ntpd as they use the same socket)

Below is my ntpd.conf without the comments:
server x.x.x.1
server x.x.x.2
driftfile       /var/lib/ntp/ntp.drift
restrict default ignore
Ntpd is running as root so there is no lack of permissions.

Both ntp servers are second accurate and their time is identical down to the split second. So why aren't the unix boxes keeping time using ntpd?