libc5 app running on kernel 2.6.25
We have a legacy linker/loader that uses libc5, and due to several factors we only have the binary and not the source. Yes, version control would have saved us from our current problem... that is now in use for our full tool chain and product line, but this particular horse is long gone. This linker runs on a linux desktop (32 bit) PC, and produces code to run on a proprietary imbedded RISC CPU.
This linker works when run on linux kernel 2.6.24, but with 2.6.25 (and 2.6.26) it fails with the message
Virtual memory exceeded in `new'
We had a similar problem with the corresponding legacy compiler, but with some research have discovered that the compiler problem was caused by the "brk randomization" in linux kernel 2.6.25. The workaround for that is to set a sysctl var and an environment var:
/proc/sys/kernel/randomize_va_space = 0 or 1
setenv MALLOC_TOP_PAD_ 536870912
This, however, does not help the linker.
I have found from using "ldd" that the linker has more shared library dependencies (the compiler only had the libc.so.5):
libg++.so.27 => /usr/lib/libg++.so.27 (0xb7eca000)
libstdc++.so.27 => /usr/lib/libstdc++.so.27 (0xb7e99000)
libm.so.5 => /lib/libm.so.5 (0xb7e90000)
libc.so.5 => /lib/libc.so.5 (0xb7dd3000)
And I have read that I may have to install the libc5 version of libg++.so.27. I hesitate to do that since I dont know if that will override the latest libg++.so.27 and cause problems for non-libc5 apps.
So, do I find and install the libc5 version of libg++.so.27, or is there some better way to disable brk randomization, or is there another difference between kernel 2.6.24 and 2.6.25 that is causing the linker problem?