Linux giving timeout for autentication

[Zarnick]

Hello, I have a server for running Oracle that from times to times it simply gives timeouts when I try to log in, via SSH and via local console! Even hitting the button for shutdown via ACPI won't work, I have to hold the button for shuttind down, and the server get's inconsistent file system and all.

I wonder if anyone could help me trying to find why this is happening (/var/log/messages shows nothing really usefull...).

Thank you.

[Irithori]

It is possible, that the machine is swapping.
That means it runs out of memory and uses the swap partition.

If that happens a lot and continous, the machine is barely useable and very slow.
Also, if even the swap partition is not big enough,
the kernel will start killing processes to get some free memory.

If you have a trend monitoring tool like Munin (there are many alternatives) you can look how the ressources are used over time.

Example from a demo installation:
Munin :: ping.uio.no :: knoll.ping.uio.no :: memory

[Zarnick]

Hum, tough it makes sense it's kinda hard to believe, it has 8GB physical RAM and 1.6Gb of Swap.

Any other opinions?

[Irithori]

Depending on what you do with oracle and how it is configured, 8GB might not be that much.

I would rule out swapping first, as it matches your problem description.

[Rubberman]

On systems with a lot of disc write activity it can become I/O bound unloading the write-behind cache to disc. This can be mitigated somewhat by forcing output to disc on a regular basis with the sync command. On systems with a lot of RAM, the OS will allocate a lot of space to disc cache so that for the most part file system performance is greatly enhanced, but as I said, the downside is when it has a lot of data to write to disc and AFAIK that is an atomic operation.

[Zarnick]

Hum, and how can I check this?
==EDIT==
We do have a memory/cpu network monitoring tool, but since Oracle always takes up almos all free memory (physical) it's hard to say if it used everything (always peeking), and both CPU and Swap we have no alerts or strangeness...

[Irithori]

Which also would be nice to have in a trend graph.
Just to know what is going on.

But for a short term, watching "vmstat 1" and "atop 1" might give some clues.

[Zarnick]

Here's the result for vmstat 1

vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 0 4436204 101788 982152 0 0 6 7 165 174 1 0 98 0
0 0 0 4436180 101788 982152 0 0 0 0 1118 3498 0 0 100 0
0 0 0 4436172 101788 982152 0 0 0 28 1065 3443 0 0 100 0
0 0 0 4436172 101788 982152 0 0 0 0 1047 3507 1 0 98 0
0 0 0 4436300 101788 982152 0 0 0 68 1123 3622 1 1 98 0
0 0 0 4436236 101800 982140 0 0 0 72 1110 3589 0 0 100 0
0 0 0 4436236 101800 982140 0 0 0 0 1170 3431 0 0 100 0
0 0 0 4436220 101800 982140 0 0 0 120 1062 3347 0 0 100 0

[Irithori]

Judging from that short log, the machine does nothing.
No swapping, no disk IO, no CPU load.

Do it again, while the machine is under stress.

Ah ok, does your tool also log disk IO?

[Zarnick]

As I said, we do have the monitoring, but it shows up nothing, the machine really is most of the part idle (it's a test machine), and sometimes we just can't log in, it gives timeout...