Hello. I am new to Linux administration. We suspect that we had a security breach on our servers, and I am assisting in the investigation.
Some background. The server is running the SQUID program and is functioning as a web proxy. Last week, squid stopped running and wouldn't be restarted. I found out that the log files had become too large, and I rotated them. Once this was done, I was able to get the server back on line. After looking through the logs of squid, it was obvious that the files were previously being rotated every sunday night at 02:59:00. After digging, I found that the server had a cronjob set to run the squidstats program at this time every week. I reviewed the cron logs and could see that this rotation was indeed happening up until a week ago. This is an excerpt from the log where the log rotation was working.
Mar 26 01:01:05 webproxy CROND: (root) CMD (run-parts /etc/cron.hourly)
Mar 26 02:00:00 webproxy CROND: (root) CMD (rdate -s *edited out*) )
Mar 26 02:01:04 webproxy CROND: (root) CMD (run-parts /etc/cron.hourly)
Mar 26 02:59:00 webproxy CROND: (root) CMD (/usr/bin/squidstats2)
Mar 26 03:00:00 webproxy CROND: (root) CMD (rdate -s *edited out*)
The next rotation was scheduled for April 2nd, but on April 2nd, it did not happen. This caused the logs to fill up and the server to stop. Here is an excerpt from the log on April 2nd.
Apr 2 01:00:00 webproxy CROND: (root) CMD (rdate -s *edited out*)
Apr 2 01:01:04 webproxy CROND: (root) CMD (run-parts /etc/cron.hourly)
Apr 2 03:00:00 webproxy CROND: (root) CMD (rdate -s *edited out*)
You can see that no cronjob was run between 01:01:04 and 03:00:00. You can also see that the cron process started working again immediately after it missed the job to run the squid rotation.
Now, the cronjob was still present when I reviewed the cronjob file. The server was not off during this time. The cron log did not show that the cron process was killed or restarted.
I am not sure how this process was stopped and restarted without being logged. Is there any way to find out. And is there anyway to trace who might have done this?
I appreciate any help.