Duplicate logs in /var/log/messages

[solidsnake1298]

I am having an issue where all the logs for my server are written twice consecutively in /var/log/messages. Because the application running on this server requires a lot of disk I/O having duplicate entries written to disk is causing a lot of events to be dropped due to the unnecessary load. I have increased the queue depth and increased the priority_boost, which has reduce the number of dropped events from about 20k/hour to less than 700/hour, but stopping the duplicate entries would solve the problem altogether.

I have read about suppressing duplicate messages, but that solution appears to only apply to messages I define. Since ALL messages are written twice this would not work for me.

Any help would be appreciated.

[Irithori]

Hi and welcome

can you please tell the OS and OS version, as well as the syslog daemon implementation (syslog, syslog-ng, rsyslog) you are using?

From the top of my head, maybe there is an overlapping config.
ie: The facility and priority your application is using is picked up by two config lines with the same logfile as target.
A snippet of the relevant config file(s) would help.

Other than that:
If your app requires a lot of IO and logging is high enough to influence it, then it makes sense to decouple app and logging by using different spindles/raids.

[solidsnake1298]

I am running Redhat 5.9 and I am running the syslog implementation.

I just had my SA send me the syslog.conf file and here it is without comments.

*.info;mail.none;authpriv.none;cron.none	/var/log/messages

authpriv.*  			/var/log/secure

mail.*       			-/var/log/maillog

cron.*      			/var/log/cron

*.emerg    			*

uucp,news.crit			/var/log/spooler

local7.*			/var/log/boot.log

auth,user.*			/var/log/messages

auth.user.*;auth,user.!=debug	@loghost

kern.*;kern.!=debug		@loghost

daemon.*			/var/log/daemon.log

syslog.*			/var/log/syslog

lpr,news,uucp,local10,local1.*	/var/log/unused.log

local2,local3,local4,local5,local6.*	/var/log/unused.log

[Irithori]

What facility/severity is your application sending to?
From the config, e.g. user.info might potentially be responsible for the duplicates.

Which could be easily fixed by introducing and using e.g. local0.* and an app specific logfile /var/log/app.log exclusively for your application.

[solidsnake1298]

Our application does not send anything to syslog. It just generates a lot of disk I/O. If it helps at all, a lot of the duplicates events we see are of type "CRED_DISP," "USER_END," "USER_START," etc.