malloc() only works under the gdb debugger.

[Unicks]

I am trying to run SCO OSR5 binaries on FC10 using the ibcs-3.8 patch. Programs like /bin/cat copied from the SCO box work fine but my RDMS(SeaChange) all fail with the following:

sh-3s.2# sgen
S_strsave(3215683390) cannot allocate space
sh-3s.2# sgen

If i use 'ltrace sgen' then i get the following (not all output shown but very near start of code):

close(499) = -1
getenv("SP") = "/usr"
strlen("/usr") = 4
malloc(5) = NULL
vfprintf(0x80fd1b8, "S_strsave(%lu) cannot allocate s"..., 0xbf80e3a0S_strsave(3212906310) cannot allocate space) = 43
strrchr("S_strsave(%lu) cannot allocate s"..., '\n') = NULL
fprintf(0x80fd1b8, "\n"
) = 1
_exithandle(0xbf80e370, 0x80cf7df, 1, 0x80fb974, 0x55f328 <unfinished ...>
_cleanup(0x80fb974, 0xbf80e554, 0x5214f0, 0xbf80e370, 0x80df22a) = 0
<... _exithandle resumed> ) = 0x5590c0
_cleanup(0xbf80e370, 0x80cf7df, 1, 0x80fb974, 0x55f328 ) = 0
_exit(1 <unfinished ...>
+++ exited (status 1) +++

It is malloc() that fails.

If i run the app (sgen) under the debugger (gdb) it works ok!!
Taking a step further, and running both ltrace and sgen under gdb then the ltrace output is like this:

close(499) = -1
getenv("SP") = "/usr"
strlen("/usr") = 4
malloc(5) = 0x80ff5bc
strcpy(0x80ff5bc, "/usr") = 0x80ff5bc
strrchr("sgen", '/') = NULL
sprintf("/usr/seachange/lib/lang/", "%s%c%s%clib%clang%c", "/usr", '/', "seachange", '/', '/', '/') = 24
getenv("SCLANG") = NULL
strcat("/usr/seachange/lib/lang/", "English") = "/usr/seachange/lib/lang/English"
fopen("/usr/seachange/lib/lang/English", "r") = 0x80fd278
fread(0xbffff22c, 1, 16, 0x80fd278 ) = 16
calloc(1, 516) = 0x80ffb40
calloc(1, 20) = 0x80ffd48
calloc(1, 20) = 0x80ffd60
fread(0xbffff1d8, 1, 4, 0x80fd278 ) = 4
malloc(8 ) = 0x80ff5c8
fread(0xbffff1d8, 1, 8, 0x80fd278 ) = 8

So why does malloc() only work under the debugger???

Is it memory protection, permissions or what?

[Rubberman]

A quick swag would be that the libc used by the application and that used by the debugger are different. Since you are running SCO binaries, I would assume that you are also using SCO shared libraries with your SCO applications. Make sure they have the correct permissions set on them.

[Unicks]

Its a thought RM, but if you remove /usr/lib/libc.so.1 (the SCO library), the both situations fail with:
libc.so.1: Bad ELF interpreter: No such file or directory

Is there any way to get more out of the debugger? I'm not familiar with it.

[Rubberman]

Are you running gdb, the Linux GNU debugger? Also, is your host system a 32-bit or 64-bit Linux OS?