system hang

[gordonthecat]

New member. Tried searching for 'Suse system hang' and run-crons and "run-crons" but no joy so aoplogies if this isn't unique.

Our Suse server hung today while I was out of the office. They couldn't shut it down or get the database restarted after nightly backup so I got them to power down.

People could log in but the Cache database (Intersystems) had not re-started from the nightly backup.

Looking in /tmp run-crons* files have appeared from Oct 17th and then stopped after the system re-start. We have been running the same crontab file for ages so I don't understand where these have come from and if this is anything to do with my system hang.

From reading elsewhere it would seem that run-crons should run every 15 minutes. The files on our system were every hour. see below

No idea where to start at all here. Going on holiday for a week tomorrow so if I have a fault I need to try and fix it or I'll spend all week on the phone.

ls -ltr in /tmp give this from 15th Oct. No changes applied so where do the run-crons come from?

-rw-r--r-- 1 root root 0 Oct 15 07:00 NSTSerr20091015
-rw-r--r-- 1 root root 86 Oct 15 07:15 NSTS20091015
-rw-r--r-- 1 root root 0 Oct 15 19:00 status100err20091015
-rw-r--r-- 1 root root 51 Oct 15 19:00 status10020091015
-rw-r--r-- 1 root root 0 Oct 15 19:00 fix_nnn_spaceerr20091015
-rw-r--r-- 1 root root 858 Oct 15 19:00 fix_nnn_space20091015
-rw-r--r-- 1 root root 0 Oct 15 21:30 backuperr20091015
-rw-r--r-- 1 root root 523 Oct 15 21:36 backup20091015
-rw-r--r-- 1 root root 0 Oct 16 19:00 status100err20091016
-rw-r--r-- 1 root root 51 Oct 16 19:00 status10020091016
-rw-r--r-- 1 root root 0 Oct 16 19:00 fix_nnn_spaceerr20091016
-rw-r--r-- 1 root root 17 Oct 16 19:00 fix_nnn_space20091016
-rw-r--r-- 1 root root 0 Oct 16 21:30 backuperr20091016
-rw-r--r-- 1 root root 523 Oct 16 21:35 backup20091016
-rw-r--r-- 1 root root 0 Oct 16 21:35 backup.end
drwx------ 2 root root 48 Oct 17 13:45 ssh-rJhGsy7872
drwx------ 2 root root 48 Oct 17 13:45 run-crons.v29946
drwx------ 2 root root 48 Oct 17 13:45 run-crons.i31130
drwx------ 2 root root 48 Oct 17 13:45 run-crons.h30414
drwx------ 2 root root 48 Oct 17 13:45 run-crons.a29681
drwx------ 2 root root 48 Oct 17 13:45 run-crons.W30659
drwx------ 2 root root 48 Oct 17 13:45 run-crons.T31456
drwx------ 2 root root 48 Oct 17 13:45 run-crons.P30886
drwx------ 2 root root 48 Oct 17 13:45 run-crons.M30188
drwx------ 2 root root 48 Oct 17 13:45 run-crons.L32423
drwx------ 2 root root 48 Oct 17 13:45 run-crons.H29452
drwx------ 2 root root 48 Oct 17 13:45 run-crons.E30005
-rw-r--r-- 1 root root 108909 Oct 17 13:45 oldbackup.tar.gz
drwx------ 2 root root 48 Oct 17 13:45 gpg-SIHcvL
-rw-r--r-- 1 root root 0 Oct 17 19:00 fix_nnn_spaceerr20091017
-rw-r--r-- 1 root root 0 Oct 17 19:00 fix_nnn_space20091017
drwx------ 2 root root 80 Oct 17 20:00 run-crons.a20273
drwx------ 2 root root 80 Oct 17 21:00 run-crons.R20517
drwx------ 2 root root 80 Oct 17 22:00 run-crons.T20739
drwx------ 2 root root 80 Oct 17 23:00 run-crons.U20983
drwx------ 2 root root 80 Oct 18 00:00 run-crons.B21226
drwx------ 2 root root 80 Oct 18 01:00 run-crons.D21451
drwx------ 2 root root 80 Oct 18 02:00 run-crons.M21694
drwx------ 2 root root 80 Oct 18 03:00 run-crons.Z21914
drwx------ 2 root root 80 Oct 18 04:00 run-crons.f22177
drwx------ 2 root root 80 Oct 18 04:15 run-crons.G22237
drwx------ 2 root root 80 Oct 18 05:00 run-crons.H22417
drwx------ 2 root root 80 Oct 18 06:00 run-crons.T22637
drwx------ 2 root root 80 Oct 18 07:00 run-crons.u22881
drwx------ 2 root root 80 Oct 18 08:00 run-crons.m23102
drwx------ 2 root root 80 Oct 18 09:00 run-crons.F23343
drwx------ 2 root root 80 Oct 18 10:00 run-crons.r23582
drwx------ 2 root root 80 Oct 18 11:00 run-crons.I23806
drwx------ 2 root root 80 Oct 18 12:00 run-crons.l24048
drwx------ 2 root root 80 Oct 18 13:00 run-crons.c24269
drwx------ 2 root root 80 Oct 18 14:00 run-crons.w24511
drwx------ 2 root root 80 Oct 18 15:00 run-crons.W24754
drwx------ 2 root root 80 Oct 18 16:00 run-crons.C24976
drwx------ 2 root root 80 Oct 18 17:00 run-crons.O25218
drwx------ 2 root root 80 Oct 18 18:00 run-crons.w25438
drwx------ 2 root root 80 Oct 18 19:00 run-crons.K25687
-rw-r--r-- 1 root root 0 Oct 18 19:00 fix_nnn_spaceerr20091018
-rw-r--r-- 1 root root 403 Oct 18 19:00 fix_nnn_space20091018
drwx------ 2 root root 80 Oct 18 20:00 run-crons.j25938
drwx------ 2 root root 80 Oct 18 21:00 run-crons.k26157
drwx------ 2 root root 80 Oct 18 22:00 run-crons.f26400
drwx------ 2 root root 80 Oct 18 23:00 run-crons.h26619
drwx------ 2 root root 80 Oct 19 00:00 run-crons.e26862
drwx------ 2 root root 80 Oct 19 01:00 run-crons.f27106
drwx------ 2 root root 80 Oct 19 02:00 run-crons.P27329
drwx------ 2 root root 80 Oct 19 03:00 run-crons.R27571
drwx------ 2 root root 80 Oct 19 04:00 run-crons.E27802
drwx------ 2 root root 80 Oct 19 04:15 run-crons.v27867
drwx------ 2 root root 80 Oct 19 05:00 run-crons.L28048
drwx------ 2 root root 80 Oct 19 06:00 run-crons.w28291
drwx------ 2 root root 80 Oct 19 07:00 run-crons.d28511
drwx------ 2 root root 80 Oct 19 08:00 run-crons.J28757
drwx------ 2 root root 80 Oct 19 09:00 run-crons.f29034
drwx------ 2 root root 80 Oct 19 10:00 run-crons.G29351
drwx------ 2 root root 80 Oct 19 11:00 run-crons.M29733
drwx------ 2 root root 80 Oct 19 12:00 run-crons.X30064
drwx------ 2 root root 80 Oct 19 13:00 run-crons.R30358
drwx------ 2 root root 80 Oct 19 14:00 run-crons.B31060
drwx------ 2 root root 80 Oct 19 15:00 run-crons.l31487
drwx------ 2 root root 80 Oct 19 16:00 run-crons.f31896
drwx------ 2 root root 80 Oct 19 17:00 run-crons.D32249
drwx------ 2 root root 80 Oct 19 18:00 run-crons.w32514
-rw-r--r-- 1 root root 0 Oct 19 19:00 status100err20091019
-rw-r--r-- 1 root root 51 Oct 19 19:00 status10020091019
-rw-r--r-- 1 root root 0 Oct 19 19:00 status100.start
-rw-r--r-- 1 root root 0 Oct 19 19:00 status100.end
drwx------ 2 root root 80 Oct 19 19:00 run-crons.g32760
-rw-r--r-- 1 root root 0 Oct 19 19:00 fix_nnn_spaceerr20091019
-rw-r--r-- 1 root root 1138 Oct 19 19:00 fix_nnn_space20091019
-rw-r--r-- 1 root root 0 Oct 19 19:00 fix_nnn_space.start
-rw-r--r-- 1 root root 0 Oct 19 19:00 fix_nnn_space.end
drwx------ 2 root root 80 Oct 19 20:00 run-crons.rhK549
drwx------ 2 root root 80 Oct 19 21:00 run-crons.edM793
-rw-r--r-- 1 root root 0 Oct 19 21:30 backuperr20091019
-rw-r--r-- 1 root root 0 Oct 19 21:30 backup.start
-rw-r--r-- 1 root root 130 Oct 19 21:35 backup20091019
drwx------ 2 root root 80 Oct 19 22:00 run-crons.GM1119
drwx------ 2 root root 80 Oct 19 23:00 run-crons.Lm1365
drwx------ 2 root root 80 Oct 20 00:00 run-crons.OA1586
drwx------ 2 root root 80 Oct 20 01:00 run-crons.mZ1833
drwx------ 2 root root 80 Oct 20 02:00 run-crons.CK2081
drwx------ 2 root root 80 Oct 20 03:00 run-crons.Pz2302
drwx------ 2 root root 80 Oct 20 04:00 run-crons.be2575
drwx------ 2 root root 80 Oct 20 04:15 run-crons.Xh2634
drwx------ 2 root root 80 Oct 20 05:00 run-crons.Ny2795
drwx------ 2 root root 80 Oct 20 06:00 run-crons.PW3039
drwx------ 2 root root 80 Oct 20 07:00 run-crons.aa3283
drwx------ 2 root root 80 Oct 20 08:00 run-crons.lS3521
drwx------ 2 root root 80 Oct 20 09:00 run-crons.Gv3768
drwx------ 2 root root 80 Oct 20 10:00 run-crons.Mm4146
drwx------ 3 root root 128 Oct 20 10:16 kde-root
drwx------ 3 root root 112 Oct 20 10:21 mcop-root
drwx------ 2 root root 48 Oct 20 10:21 ksocket-root
drwxrwxrwt 2 root root 48 Oct 20 10:21 .ICE-unix
drwxr-xr-x 25 root root 648 Oct 20 10:24 ..
-rw------- 1 root root 0 Oct 20 10:25 OW_LCL_APIIPC_72859_Xq47Bf_P9r7
61-5_J-7_Q.lock
srw-rw-rw- 1 root root 0 Oct 20 10:25 OW_LCL_APIIPC_72859_Xq47Bf_P9r7
61-5_J-7_Q
-r--r--r-- 1 root root 11 Oct 20 10:25 .X0-lock
drwxrwxrwt 2 root root 72 Oct 20 10:25 .X11-unix
drwx------ 4 root root 96 Oct 20 10:25 0252521392
drwxrwxrwt 100 root root 10704 Oct 20 14:15 .

[robertdaleweir]

Hi gordonthecat
Can you, as root, post the results of

crontab -l

. All cron jobs should be set out in the crontab file, which this will show. It appears that a lock has been put on X, probably in <.X0-lock> . This lock file may have to be removed, first made writable by root. Just post contents of <.X0-lock> Use <cat .X0-lock> and post results here. Not sure I can help but lets see. Cheers...
Robert

[coopstah13]

Could be running out of disk space if your cron jobs are generating lots of logging output.

[gordonthecat]

Plenty disk space. Checked that first.

trane:/tmp # crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.12137 installed on Thu Sep 10 15:07:41 2009)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
# Root crontab table
30 21 * * 1-5 /usr/local/bin/run_backup_trane
0 07 1,15 * * /usr/local/bin/NSTSTrace
45 13 17 * * /usr/local/bin/rmtmp.sh
40 12 18 2,5,8,11 * /usr/local/bin/rmapplic.sh
00 19 * * 1-5 /usr/local/bin/run_status100
00 19 * * 0-6 /usr/local/bin/run_fix_nnn_space

and
trane:/tmp # cat .X0-lock
7660
trane:/tmp #

Thanks Robert

[robertdaleweir]

Hi gordonthecat
First remove .X0-lock as root. Second will the backup run from a console as root after you erase the lock file (remember you will have to make the lock file writeable if you want to erase it, as root or whatever user it belongs to). If so at least you will know that the backup will run and the problem is elsewhere.
According to the crontab your <backup> is set to run every day of the work week Monday through Friday at 21:30 hours. </usr/local/bin/run_backup_trane>. Not sure about the other items but I assume that the backup item is where your problems are, as per your information.
The cron jobs (crond) are scanned at least every minute by the system when <crond> is running as a service. It is running, in your case, as you would not get any info from <crontab -l> if it were not.
If you can run the backup interactively then someone should restart your <crond> service as root. <service crond restart>. I would think that the problem is with the Database (you mentioned) and perhaps it is corrupted and needs to be taken down and restarted (possibly rolled back or something) as well.
I am sorry I could not get back to you sooner but I was away yesterday. Hope this helps in some way. Cheers...
Robert