I'd say go with a lower kernel and work your way up.
Otherwise, it's about specializing the kernel for the computer.
Linux From Scratch seems like a really good idea.
MuLinux -> Damn Small Linux -> Puppy Linux -> Debian
I'm still hoping for a day when tons of programs will be created with ncurses, and that idea of an X server will seem silly. After using Linux for a while, I can see why people argue about the X server.
I'm sure most oldbies truly think all this graphical stuff is nonsense, and it could be simplified down somewhat.
In my view, the Xserver is only necessary for viewing pictures. I mean, if you remove the idea of graphic-intense games from your head. With that in mind, that's how I think of putting Linux on an old computer. The OS shouldn't bog the computer, and it should run decently fast. I'd say Damn Small Linux's programs do well in that department. DSL is limited, but serves its purpose.
I keep reading about web browsers that can be text-based, like lynx or w3m, and they use some kind of
framebuffer if you need to look at an image. I guess the idea is saving on system resources and using them only when you really need to. Of course, the less system resources you use, the faster you can have the computer.
FrameBuffer UI - Wikipedia, the free encyclopedia