Unfortunately I won't help you but i have the same problem, i followed the instructions in readme and recieve the same error : Kernel Panic. i think it could be a configuration problem (in make menuconfig) ,but tried many possibilities .and the problem still occurs. did you try configuration file from the site you entered ?

i have Debian ,kernel 2.2.20 and my USB optical mouse isn't working,then a i cannot run Xwin .i don't know what to put in XF86Config-4 at this time it is
option "Protocol" "ImPS/2"
option "Device" "dev/input/mice"

if i put "Protocol" "usb" i recieve "unknown protocol" ,
i have XFree86 4.5.0