My server isn't really that powerful, dual PII processors so I don't use a graphical interface. I run it all from text mode but I still use SuSE, if I need any extra packages then I can just get them with ease. I do a complete installation from an ftp site and then get any extra programs from there.
I have to use SuSE 9.1 as the later versions don't pick up my SCSI raid card, oh well.
I find SuSE doesn't interfere with what you are doing either so you actually have full control of the system. If you want to see what it is running then just look at the link in my sig below.