I remember that when I used to use XFree87, there was a thing called the X-font server (xfs) that ran as a background daemon. All the writing on X depended on it. There doesn't seem to be anything like that now. Does xserver-xorg get its fonts directly from the fonts directory under /usr/share? I'm just curious.