Any open of a socket will fail with ENXIO, e.g. running "</tmp/.X11-unix/X0" in sh will produce No such device or address. If you want to check the details, pass it to sh via -c and strace it (or just strace cat /tmp/.X11-unix/X0).
I checked that ln thing, but I think that only applies to other unices, since link(2) says "EPERM oldpath is a directory". I tested just to be sure, of course, but it didn't work. rm(1) contains a similar referense to the possibility of removing non-empty directories.