Unable to push RHEL 6.0 past 95% utilization

[rtaheri]

Some particulars:
- I am running with Oracle UEL 6.0 (2.6.32-100.28.5.el6) because stock RHEL 6.0 (2.6.32-71.el6 ) has issues with the async I/O driver.
- The test is a high throughput performance benchmark running on Oracle 11gR1
- I am pumping a lot of disk I/O through the system while running with enough users to max out the 8 CPUs, which get to 99+% utilization with RHEL 5.3
- The server is a 4-socket Nehalem EX X7560. Right now, only 2 cores per socket are enabled.
- There are GBs of memory left over. The disk response time of the SSD arrays is around 1ms and the arrays are capable of 4-5 times more IOPS. Same with networking, etc. The testbed is capable of maxing out 32 CPUs with RHEL 5.3

No matter what I do, I cannot push the RHEL 6.0 CPU utilization past 95-96%. The same test on all flavors of RHEL from 4.4 to 5.3 can totally saturate the CPUs. It feels like the system is intentionally holding back some CPU cycles. For what??? Is it a new scheduler in 6.0, something tied to power management, ...?

Thanks,
Reza

[Rubberman]

Good question. I am running Scientific Linux 6 (2.6.32-71.24.1 kernel) and I can easily hit 99%+ cpu utilization (dual quad core Penryn processors) on all 8 cores. Have you posted your question to Oracle?

[rtaheri]

Good question. I am running Scientific Linux 6 (2.6.32-71.24.1 kernel) and I can easily hit 99%+ cpu utilization (dual quad core Penryn processors) on all 8 cores. Have you posted your question to Oracle?

Yes, it could be Oracle. I tried a simple soaker program. I can absorb the remaining 5% idle on all cores with the benchmark taking 95%. So something is holding the benchmark itself from taking that last 5%. It could be the DBMS, but the same exact DBMS and benchmark have had no problems saturating the system with earlier Linux versions. I am wondering whether, for example, the scheduler doesn't let a single uid or a single executable go past 95%. I have tried various tricks (timeshare versus real time priority, more users, etc.), and always hit a wall at 95%.