System:
ARM11, total memory is 240956K, Linux kernel 2.6.32.

Problem:
LTP test msgctl11 failed to fork() in Linux kernel 2.6.32 with the following message:
msgctl11 1 TFAIL : Fork failed (may be OK if under stress)
msgctl11 0 TWARN : Fork failure in first child of child group 123
msgctl11 0 TWARN : Fork failure in first child of child group 132
msgctl11 0 TWARN : Fork failure in first child of child group 137
msgctl11 0 TWARN : Fork failure in first child of child group 133
msgctl11 0 TWARN : Fork failure in first child of child group 134
msgctl11 0 TWARN : Fork failure in first child of child group 118
msgctl11 0 TWARN : Fork failure in first child of child group 96
msgctl11 0 TWARN : Fork failure in first child of child group 113
msgctl11 0 TWARN : Fork failure in first child of child group 130
msgctl11 0 TWARN : Fork failure in first child of child group 148
msgctl11 0 TWARN : Fork failure in first child of child group 153
msgctl11 0 TWARN : Fork failure in first child of child group 155

However, msgctl11 ran successfully in Linux kernel 2.6.27.18.

About msgctl11:
The basic idea of msgctl11 is to get and manipulate a message queue.
It has 3 parameters: #nreps, #nprocs, #nkids.
The program creates #nprocs processces, which then create #nkids kid processes, which will read & write the message queue for #nreps time.
In our system, for kernel 27 and 32, the default parameters are: #nreps=1000, #nprocs=470, #nkids=10.
For kernel 32, msgctl11 1000 150 10 runs well.
When #nprocs > 150, msgctl11 may fail.
If that is the case (hopefully not), it is quite bad because kernel 32 can only support about 1/3 concurrent processes of kernel 27.

What we have done:
1. Added printf to print errno, which is 12 (Cannot allocate memory)
2. Use an endless while loop to run “free” and record the result.
At the peak time, memory is still available:
total used free shared buffers
Mem: 240956 200304 40652 0 0
Swap: 0 0 0
Total: 240956 200304 40652

Did anyone have this problem before?
Any suggestions/comments will be highly appreciated.

Thanks in advance!