Help! imapd is spawning multiple processes, all of a sudden, for no reason!

[fishsponge]

Hi All,

I need some assistance, if possible...

Our IMAP server has recently (as of 10:30 GMT today) started spawning multiple processes for no reason! This is causing the mail server's load average to increase continually until the whole machine grinds to a halt.

Here is a typical scenario...

• user1 opens his mail client, which connects to IMAP. "inetd" starts up a single "imapd" process which is run as user1.
• user2 then opens his mail client, which connects to IMAP. "inetd" starts up another single "imapd" process which is run as user2.
• 60 seconds later (roughly), another imapd process is started up, running as user1, so user1 now has two imapd processes running.
• another minute or so later, user1 gains another imapd process.
• then user3 connects, and gains one process, but user2 then gains another process as well.
• This continues until all 20 users or so are connected, and they each have 8 or so processes (at which point the server grinds to a halt, and I have to reboot it).

I must strongly point out that this is seemingly random too. It is not tied to any particular user, we have not recently run any updates to the software, we have not recently changed the configuration, no users have *really* large inboxes or IMAP folders and the whole procedure happens differently every time. Different users connect, and they gain imapd processes at different rates, but the server always keeps spawning new processes and always grinds to a halt, within 3-5 minutes.

This hasn't happened for the entire 117 days that the server has been up, but at just after 10:30 today, it has started happening continually.

I've even tried rebooting the whole machine, and it just starts happening again!

/var/log/messages contains nothing useful at all, and all I can see in /var/log/syslog that relates to IMAP is as follows:

Feb 26 12:23:32 mail imapd[3401]: connect from 192.168.136.11 (192.168.136.11)
Feb 26 12:23:33 mail imapd[3401]: imaps SSL service init from 192.168.136.11
Feb 26 12:23:33 mail imapd[3401]: Authenticated user=username1 host=[192.168.136.11]
Feb 26 12:23:34 mail imapd[3404]: connect from 192.168.144.22 (192.168.144.22)
Feb 26 12:23:34 mail imapd[3404]: imaps SSL service init from 192.168.144.22
Feb 26 12:23:34 mail imapd[3404]: Authenticated user=username2 host=l10-2.domain.com [192.168.144.22]
Feb 26 12:23:38 mail imapd[3405]: connect from 192.168.144.104 (192.168.144.104)
Feb 26 12:23:38 mail imapd[3405]: imaps SSL service init from 192.168.144.104
Feb 26 12:23:39 mail imapd[3405]: Authenticated user=username3 host=tecra2.domain.com [192.168.144.104]
Feb 26 12:23:40 mail imapd[3409]: connect from 192.168.132.234 (192.168.132.234)
Feb 26 12:23:40 mail imapd[3409]: imap service init from 192.168.132.234
Feb 26 12:23:40 mail imapd[3409]: Authenticated user=username4 host=AlexDixon.domain.com [192.168.132.234]
Feb 26 12:23:50 mail imapd[3422]: connect from 192.168.144.46 (192.168.144.46)
Feb 26 12:23:51 mail imapd[3422]: imaps SSL service init from 192.168.144.46
Feb 26 12:23:51 mail imapd[3422]: Authenticated user=username1 host=cvg-2win.domain.com [192.168.144.46]
Feb 26 12:24:06 mail imapd[3430]: connect from 192.168.128.136 (192.168.128.136)
Feb 26 12:24:06 mail imapd[3430]: imaps SSL service init from 192.168.128.136
Feb 26 12:24:06 mail imapd[3430]: Logout user=??? host=it3.domain.com [192.168.128.136]
Feb 26 12:24:18 mail imapd[3436]: connect from 192.168.144.120 (192.168.144.120)
Feb 26 12:24:19 mail imapd[3436]: imaps SSL service init from 192.168.144.120
Feb 26 12:24:22 mail imapd[3436]: Authenticated user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:24:22 mail imapd[3445]: connect from 192.168.144.70 (192.168.144.70)
Feb 26 12:24:22 mail imapd[3445]: imaps SSL service init from 192.168.144.70
Feb 26 12:24:23 mail imapd[3445]: Authenticated user=username6 host=r500-6.domain.com [192.168.144.70]
Feb 26 12:24:29 mail imapd[3454]: connect from 192.168.144.120 (192.168.144.120)
Feb 26 12:24:29 mail imapd[3454]: imaps SSL service init from 192.168.144.120
Feb 26 12:24:29 mail imapd[3454]: Authenticated user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:24:34 mail imapd[3462]: connect from 192.168.136.11 (192.168.136.11)
Feb 26 12:24:34 mail imapd[3462]: imaps SSL service init from 192.168.136.11
Feb 26 12:24:34 mail imapd[3469]: connect from 192.168.144.36 (192.168.144.36)
Feb 26 12:24:34 mail imapd[3469]: imaps SSL service init from 192.168.144.36
Feb 26 12:24:34 mail imapd[3462]: Authenticated user=username7 host=[192.168.136.11]
Feb 26 12:24:34 mail imapd[3401]: Killed (lost mailbox lock) user=username7 host=[192.168.136.11]
Feb 26 12:24:37 mail imapd[3469]: Login user=username8 host=l10-1.domain.com [192.168.144.36]
Feb 26 12:24:37 mail imapd[3475]: connect from 192.168.144.36 (192.168.144.36)
Feb 26 12:24:37 mail imapd[3475]: imaps SSL service init from 192.168.144.36
Feb 26 12:24:38 mail imapd[3475]: Login user=username8 host=l10-1.domain.com [192.168.144.36]
Feb 26 12:24:40 mail imapd[3479]: connect from 192.168.144.104 (192.168.144.104)
Feb 26 12:24:40 mail imapd[3479]: imaps SSL service init from 192.168.144.104
Feb 26 12:24:40 mail imapd[3479]: Authenticated user=username3 host=tecra2.domain.com [192.168.144.104]
Feb 26 12:24:41 mail imapd[3485]: connect from 192.168.144.10 (192.168.144.10)
Feb 26 12:24:41 mail imapd[3485]: imaps SSL service init from 192.168.144.10
Feb 26 12:24:41 mail imapd[3485]: Authenticated user=username9 host=cvg-tecra2.domain.com [192.168.144.10]
Feb 26 12:24:41 mail imapd[3436]: Command stream end of file, while reading line user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:24:42 mail imapd[3454]: Command stream end of file, while reading line user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:24:42 mail imapd[3405]: Killed (lost mailbox lock) user=username3 host=tecra2.domain.com [192.168.144.104]
Feb 26 12:24:44 mail imapd[3490]: connect from 192.168.144.120 (192.168.144.120)
Feb 26 12:24:44 mail imapd[3490]: imaps SSL service init from 192.168.144.120
Feb 26 12:24:47 mail imapd[3490]: Authenticated user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:24:52 mail imapd[3516]: connect from 192.168.144.89 (192.168.144.89)
Feb 26 12:24:52 mail imapd[3516]: imaps SSL service init from 192.168.144.89
Feb 26 12:24:52 mail imapd[3516]: Authenticated user=username10 host=opti12.domain.com [192.168.144.89]
Feb 26 12:24:54 mail imapd[3519]: connect from 192.168.144.46 (192.168.144.46)
Feb 26 12:24:54 mail imapd[3519]: imaps SSL service init from 192.168.144.46
Feb 26 12:24:54 mail imapd[3519]: Authenticated user=username1 host=cvg-2win.domain.com [192.168.144.46]
Feb 26 12:24:54 mail imapd[3422]: Killed (lost mailbox lock) user=username1 host=cvg-2win.domain.com [192.168.144.46]
Feb 26 12:25:14 mail imapd[3552]: connect from 192.168.144.120 (192.168.144.120)
Feb 26 12:25:14 mail imapd[3552]: imaps SSL service init from 192.168.144.120
Feb 26 12:25:14 mail imapd[3552]: Authenticated user=username5 host=mkc14.domain.com [192.168.144.120]
Feb 26 12:25:19 mail imapd[3475]: Logout user=username8 host=l10-1.domain.com [192.168.144.36]

Does anyone have any idea what is going on here??

Thanks in advance!

[Ziplock]

Hi there,

Sounds to me like the previous imapd isn't finishing. This is just a hunch, but have you filled any partitions with data? I've had servers that just start behaving weirdly, and is generally turns out that my /var/ or /home etc are full. Deleting old/unwanted files may fix it. df -h