Troubleshooting When Cron Job wont Executed

What to check when Cron job/script wont executed? One of the possible answer is maybe the PATH declaration in your script which is not correct. Here are the sample story.

Currently I have to do application profiling in aim to analyst the posible bottleneck in application. The tools that I use like I have mentioned before is JBoss Profiler. Fortunately, this tool can be integrated easily with existing JBoss application server. JBoss profiler itself consist two modules. The first one is several jar files and configurations that is dipped into application server. The second part is small client Java application that will snapshot and produce profile report for current situation of method/function graph call from our application source code.

We have necessity to run the client snapshot application for every particular range time. The mean of this approach is when application that we profile start to run slowly in some particular module and some particular time we can dig to profile report and analyze the possible source of problem. To achieve this requirement we need some job scheduler tools. Because most of server live in Unix environment, we are very lucky because we have well known job scheduler in most Unix box. The scheduler name is Cron.

Scheduling a job with Cron is only the matter of editing crontab definition. Crontab itself is a configuration file that consist a list of job/command that will be executed for every particular time. This is the content of my Cron file. See the last line of the Cron files.

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
#* * * * * root topCat.sh
#* * * * * root iostatCat.sh
* * * * * root snapshot.sh # this will be run every minute

This is the snapshot.sh script content.

#! /bin/sh
#This script is intended to take profiling snapshot

export PATH=$PATH:/usr/local/jdk1.6.0_20/bin
cd /opt/jboss-profiler-2.0.0.Beta5
# run the profiler snapshot capturer
java -Xmx512m -Djboss-profiler-client.properties=jboss-profiler-client.properties -jar jboss-profiler-client.jar snapshot
# move the produced report to other folder, in this case is to apache html folder so we can see remotely from browser client 
mv 201107* /var/www/html/profiler

#exit 0

I have run this configuration in my local successfully. But when I clone all the setting into the server the process seems not work as expected. I became more curious because after catching the log of cron tail -f /var/log/cron (or /var/log/syslog on ubuntu) the script actually already executed. The other of my confusedness, the script itself is worked perfectly if we executed it directly on the shell.

Jul 13 11:02:01 ID31-ND102 crond[13350]: (root) CMD (snapshot.sh)
Jul 13 11:03:01 ID31-ND102 crond[13368]: (root) CMD (snapshot.sh)
Jul 13 11:04:01 ID31-ND102 crond[13397]: (root) CMD (snapshot.sh)
Jul 13 11:05:01 ID31-ND102 crond[13418]: (root) CMD (snapshot.sh)
Jul 13 11:06:01 ID31-ND102 crond[13447]: (root) CMD (snapshot.sh)
Jul 13 11:07:01 ID31-ND102 crond[13465]: (root) CMD (snapshot.sh)
Jul 13 11:08:01 ID31-ND102 crond[13493]: (root) CMD (snapshot.sh)
Jul 13 11:09:01 ID31-ND102 crond[13512]: (root) CMD (snapshot.sh)
Jul 13 11:10:01 ID31-ND102 crond[13544]: (root) CMD (snapshot.sh)
Jul 13 11:10:01 ID31-ND102 crond[13546]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Jul 13 11:11:01 ID31-ND102 crond[13564]: (root) CMD (snapshot.sh)
Jul 13 11:12:01 ID31-ND102 crond[13593]: (root) CMD (snapshot.sh)
Jul 13 11:13:01 ID31-ND102 crond[13612]: (root) CMD (snapshot.sh)
Jul 13 11:14:01 ID31-ND102 crond[13644]: (root) CMD (snapshot.sh)

Then after some hours of debugging I realize one possible source of problem is in PATH declaration in crontab files.

PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

Because I run java command then I have to add the path declaration to be like this.

PATH=/usr/local/jdk1.6.0_20/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin

Then the scheduler can run the script perfectly.

Notes: This command sometime takes long time to run. Please refer to jboss profiler documentation for more information.

java -Xmx512m -Djboss-profiler-client.properties=jboss-profiler-client.properties -jar jboss-profiler-client.jar snapshot