Help for someone who has only used the crontab in AIX

[Dan357]

Hello,
I have a PC at work running Red hat 8.0, we use this PC to run a few FTP and encryption processes for files we send to the bank.
Last Friday (11/28/11) the cron jobs stopped working and I am attempting to troubleshoot this issue. I can't think of anything that has changed.

How can I test my cron processes?

What should I be looking for as far as processes running?

I mostly work in AIX, and am not familiar with how the Linux cron works.

Thank you
Dan

[Irithori]

RedHat 8 is from 2002.
That itself wont stop a cron from running, but that system probably didnt get much attention/maintenance
for what sounds like an important part of a backend process ("encryption", "bank", "send files").

Is the assumption correct, that its hardware is also from 2002?
If yes, you might want to look for hardware errors in dmesg, /var/log/messages, local console.

Maybe the disk(s) are full?

du -hs

I didnt quite get, if you run the ftp server or the bank does.
I assume you do.
So, from another machine, check network and ftp connectivity.

ftp -v <YOUR_FTP_SERVER>

For cron itself:
1) There should be a cron process

ps auwx|grep cron

2) Check the various crontabs to find your script

cat /etc/crontab
ls -la /etc/cron.{daily,weekly,monthly,d}
ls -la /var/spool/cron/

Once you found that script:
1) If it uses a logfile or log facilty: Check the logs
2) If the script has a dry-run option, use it
3) If there is no dry-run option, but the script is safe to run multiple times: do so
4) If there is no dry-run option, and the script shall not run multiple times: Then you need to open the script and debug it and
trace each step.



P.S.: I am curious: What is the uptime of that box?

[Dan357]

Hello,
First, thanks for taking the time to answer my questions. I was feeling a little lost.

I think the network connections and encryption keys are working fine because I can run the scripts manually from the box and they work fine.

The PC is has been in use for about six years, and it hardly ever gets rebooted. I did reboot it yesterday.

1. When I ran the du -hs command from the root directory it returned
[root@pwc3 root]# du -hs
7.8M .



2. When I run the
[root@pwc3 root]# ps auwx|grep cron
root 732 0.0 0.4 1516 612 ? S Oct31 0:00 crond
root 2500 0.0 0.4 3276 624 pts/0 S 08:26 0:00 grep cron



Here is the last 50 lines of the /var/log/messages file

[root@pwc3 log]# tail messages -n 50
Nov 1 07:15:39 pwc3 ftpd[2386]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:15:39 pwc3 ftpd[2386]: FTP session closed
Nov 1 07:20:39 pwc3 ftpd[2388]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:20:39 pwc3 ftpd[2388]: FTP session closed
Nov 1 07:25:39 pwc3 ftpd[2392]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:25:39 pwc3 ftpd[2392]: FTP session closed
Nov 1 07:30:39 pwc3 ftpd[2394]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:30:40 pwc3 ftpd[2394]: FTP session closed
Nov 1 07:35:39 pwc3 ftpd[2396]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:35:39 pwc3 ftpd[2396]: FTP session closed
Nov 1 07:40:40 pwc3 ftpd[2400]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:40:40 pwc3 ftpd[2400]: FTP session closed
Nov 1 07:45:40 pwc3 ftpd[2402]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:45:40 pwc3 ftpd[2402]: FTP session closed
Nov 1 07:50:40 pwc3 ftpd[2404]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:50:40 pwc3 ftpd[2404]: FTP session closed
Nov 1 07:55:40 pwc3 ftpd[2408]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 07:55:40 pwc3 ftpd[2408]: FTP session closed
Nov 1 08:00:40 pwc3 ftpd[2410]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:00:40 pwc3 ftpd[2410]: FTP session closed
Nov 1 08:05:40 pwc3 ftpd[2414]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:05:40 pwc3 ftpd[2414]: FTP session closed
Nov 1 08:10:40 pwc3 ftpd[2418]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:10:40 pwc3 ftpd[2418]: FTP session closed
Nov 1 08:15:40 pwc3 ftpd[2422]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:15:40 pwc3 ftpd[2422]: FTP session closed
Nov 1 08:15:47 pwc3 login(pam_unix)[2421]: session opened for user campbell by
(uid=0)
Nov 1 08:15:47 pwc3 -- campbell[2421]: LOGIN ON pts/0 BY campbell FROM sa72.pw
csa.org
Nov 1 08:15:55 pwc3 su(pam_unix)[2453]: session opened for user root by campbel
l(uid=504)
Nov 1 08:20:40 pwc3 ftpd[2492]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:20:40 pwc3 ftpd[2492]: FTP session closed
Nov 1 08:25:40 pwc3 ftpd[2498]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:25:40 pwc3 ftpd[2498]: FTP session closed
Nov 1 08:30:40 pwc3 ftpd[2502]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:30:40 pwc3 ftpd[2502]: FTP session closed
Nov 1 08:35:40 pwc3 ftpd[2505]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:35:41 pwc3 ftpd[2505]: FTP session closed
Nov 1 08:40:41 pwc3 ftpd[2511]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:40:41 pwc3 ftpd[2511]: FTP session closed
Nov 1 08:42:08 pwc3 su(pam_unix)[2453]: session closed for user root
Nov 1 08:42:09 pwc3 login(pam_unix)[2421]: session closed for user campbell
Nov 1 08:42:35 pwc3 login(pam_unix)[2517]: authentication failure; logname= uid
=0 euid=0 tty=pts/0 ruser= rhost=sa72.pwcsa.org user=campbell
Nov 1 08:42:37 pwc3 login[2517]: FAILED LOGIN 1 FROM sa72.pwcsa.org FOR campbel
l, Authentication failure
Nov 1 08:42:46 pwc3 login(pam_unix)[2517]: session opened for user campbell by
(uid=0)
Nov 1 08:42:46 pwc3 -- campbell[2517]: LOGIN ON pts/0 BY campbell FROM sa72.pw
csa.org
Nov 1 08:42:52 pwc3 su(pam_unix)[2548]: session opened for user root by campbel
l(uid=504)
Nov 1 08:45:41 pwc3 ftpd[2591]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:45:41 pwc3 ftpd[2591]: FTP session closed
Nov 1 08:50:41 pwc3 ftpd[2598]: wu-ftpd - TLS settings: control allow, client_c
ert allow, data allow
Nov 1 08:50:41 pwc3 ftpd[2598]: FTP session closed
[root@pwc3 log]#

Thanks again
Dan

[atreyu]

So the script that you run from the command line is the same one that does not work when cronned? Can you post the contents of this script (with sensitive parts removed, of course)? And can you also post the contents of the cronjob? If you don't know where it is, Irithori's suggestions showed you where to look.

Also, check /var/log/cron as a possible log location.

[Dan357]

Also,
I ran the the du -hs command from the root directory it returned -
[root@pwc3 root]# du -hs
7.8M .
What is this 7.8M output mean? 7.8 Megs free?

[Dan357]

Hello,
This is the /var/log/cron

[root@pwc3 root]# cat /var/log/cron
Oct 30 04:02:47 pwc3 CROND[32109]: (root) MAIL (mailed 160 bytes of output but g
ot status 0x0045 )
Oct 30 04:22:00 pwc3 CROND[32432]: (root) CMD (run-parts /etc/cron.weekly)
Oct 30 04:22:00 pwc3 anacron[32436]: Updated timestamp for job `cron.weekly' to
2011-10-30
Oct 30 05:01:00 pwc3 CROND[6341]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 06:01:00 pwc3 CROND[6375]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 07:01:00 pwc3 CROND[6409]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 08:01:00 pwc3 CROND[6443]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 09:01:00 pwc3 CROND[6477]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 10:01:00 pwc3 CROND[6511]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 11:01:00 pwc3 CROND[6545]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 12:01:00 pwc3 CROND[6579]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 13:01:00 pwc3 CROND[6621]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 14:01:00 pwc3 CROND[6655]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 15:01:00 pwc3 CROND[6691]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 16:01:00 pwc3 CROND[6725]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 17:01:00 pwc3 CROND[6759]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 18:01:00 pwc3 CROND[6793]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 19:01:00 pwc3 CROND[6827]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 20:01:00 pwc3 CROND[6861]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 21:01:00 pwc3 CROND[6895]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 22:01:00 pwc3 CROND[6929]: (root) CMD (run-parts /etc/cron.hourly)
Oct 30 23:01:00 pwc3 CROND[6963]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 00:01:00 pwc3 CROND[6997]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 01:01:00 pwc3 CROND[7031]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 02:01:00 pwc3 CROND[7065]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 03:01:00 pwc3 CROND[7099]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 04:01:00 pwc3 CROND[7133]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 04:02:00 pwc3 CROND[7135]: (root) CMD (run-parts /etc/cron.daily)
Oct 31 04:02:03 pwc3 anacron[7292]: Updated timestamp for job `cron.daily' to 20
11-10-31
Oct 31 04:02:56 pwc3 CROND[7134]: (root) MAIL (mailed 160 bytes of output but go
t status 0x0045 )
Oct 31 05:01:00 pwc3 CROND[7475]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 06:01:00 pwc3 CROND[7509]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 07:01:00 pwc3 CROND[7543]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 08:01:00 pwc3 CROND[7577]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 09:01:00 pwc3 CROND[7611]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 10:01:00 pwc3 CROND[7723]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 10:56:25 pwc3 crontab[7759]: (root) LIST (root)
Oct 31 10:56:57 pwc3 crontab[7795]: (tto) LIST (tto)
Oct 31 12:01:00 pwc3 CROND[7980]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 12:42:43 pwc3 crontab[8115]: (tto) LIST (tto)
Oct 31 13:01:00 pwc3 CROND[8128]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 13:51:06 pwc3 crond[732]: (CRON) STARTUP (fork ok)
Oct 31 13:51:07 pwc3 anacron[774]: Anacron 2.3 started on 2011-10-31
Oct 31 13:51:07 pwc3 anacron[774]: Will run job `cron.monthly' in 75 min.
Oct 31 13:51:07 pwc3 anacron[774]: Jobs will be executed sequentially
Oct 31 14:01:00 pwc3 CROND[952]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 15:01:00 pwc3 CROND[1015]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 15:06:07 pwc3 anacron[774]: Job `cron.monthly' started
Oct 31 15:06:07 pwc3 anacron[774]: Job `cron.monthly' terminated
Oct 31 15:06:07 pwc3 anacron[774]: Normal exit (1 jobs run)
Oct 31 15:06:07 pwc3 anacron[1025]: Updated timestamp for job `cron.monthly' to
2011-10-31
Oct 31 15:42:34 pwc3 crontab[1076]: (campbell) LIST (campbell)
Oct 31 15:43:09 pwc3 crontab[1115]: (root) LIST (root)
Oct 31 15:43:49 pwc3 crontab[1147]: (tto) LIST (tto)
Oct 31 16:01:00 pwc3 CROND[1268]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 16:07:25 pwc3 crontab[1276]: (root) LIST (root)
Oct 31 16:07:38 pwc3 crontab[1312]: (tto) LIST (tto)
Oct 31 17:01:00 pwc3 CROND[1364]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 18:01:00 pwc3 CROND[1401]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 19:01:00 pwc3 CROND[1435]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 20:01:00 pwc3 CROND[1469]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 21:01:00 pwc3 CROND[1504]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 21:33:02 pwc3 crontab[1620]: (root) LIST (root)
Oct 31 21:33:11 pwc3 crontab[1652]: (tto) LIST (tto)
Oct 31 22:01:00 pwc3 CROND[1678]: (root) CMD (run-parts /etc/cron.hourly)
Oct 31 23:01:00 pwc3 CROND[1712]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 00:01:00 pwc3 CROND[1746]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 01:01:00 pwc3 CROND[1780]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 02:01:00 pwc3 CROND[1814]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 03:01:00 pwc3 CROND[1848]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 04:01:00 pwc3 CROND[1882]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 04:02:00 pwc3 CROND[1884]: (root) CMD (run-parts /etc/cron.daily)
Nov 1 04:02:03 pwc3 anacron[2041]: Updated timestamp for job `cron.daily' to 20
11-11-01
Nov 1 04:02:57 pwc3 CROND[1883]: (root) MAIL (mailed 160 bytes of output but go
t status 0x0045 )
Nov 1 04:42:00 pwc3 CROND[2214]: (root) CMD (run-parts /etc/cron.monthly)
Nov 1 04:42:01 pwc3 anacron[2218]: Updated timestamp for job `cron.monthly' to
2011-11-01
Nov 1 05:01:00 pwc3 CROND[2230]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 06:01:00 pwc3 CROND[2264]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 07:01:00 pwc3 CROND[2378]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 08:01:00 pwc3 CROND[2412]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 09:01:00 pwc3 CROND[2609]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 10:01:00 pwc3 CROND[2714]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 11:01:00 pwc3 CROND[2748]: (root) CMD (run-parts /etc/cron.hourly)
Nov 1 12:01:00 pwc3 CROND[2885]: (root) CMD (run-parts /etc/cron.hourly)
[root@pwc3 root]#

Dan

[Irithori]

Sorry about the du -hs.
It should be df -h to check if the disk is full.

The script runs fine when called manually, but not anymore from cron?
Then something *has* changed:
- disk full, check with df -h
- the crontab line (What environment is set for this cronjob, e.g.: $PATH?)
- permissions on the script
- permissions on the workdirectory/files

[Dan357]

Hi,
I ran the df -h
[root@pwc3 bin]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 4.8G 2.1G 2.5G 44% /
/dev/hda5 3.8G 33M 3.6G 1% /ftpdata
/dev/hda6 980M 22M 908M 3% /home
none 62M 0 61M 0% /dev/shm
/dev/hda3 3.8G 38M 3.6G 2% /web
/dev/hda7 2.0G 227M 1.6G 12% /arupload
[root@pwc3

I can't think of what may have changed, I am the only one with access to the box. I am home today and just logged in and ran the processes manually. worked fine.
I will keep trying to figure out what is going on.
Thanks
Dan

If you can think of anything I can test I would be happy to try it.
This runs M-F and this is the first issue I can ever remember having with it.

[atreyu]

I would suggest redirecting the output of your cronjob to a log file, so that you may catch any errors it is experiencing. For example, if it looks like this:

* * * * * /home/user/myscript.sh

change it to this:

* * * * * /home/user/myscript.sh > /tmp/cronjob.log 2>&1

Then check the contents of /tmp/cronjob.log for script output, after it has run.

[Dan357]

Hi atreyu,
I think I may be over thinking this, how can I set up a simple cron job to test? If I can get that working I will just start over.

In AIX I would just do a crontab -e but I really don't know the process in red hat 8.0, does anyone have a link to any documentation or a site with a step by step? Where do I edit the crontab is my main question?


Thanks again everyone.
Dan