Shore Up
A heart-monitor line on a bedside machine that has gone flat, tripping a bell alarm on the desk beside it.
Linux

Automate Cron Job Monitoring So a Silent Failure Pages You

Ketan Aagja8 min read
No ratings yet

Before you run this

This guide sets up a dead man's switch for a cron job: the job pings an external monitor every time it runs, and if that ping is late or reports a non-zero exit, the monitor pages you. It exists to catch the failure mode plain cron misses entirely — a job that silently never ran, or ran and failed on a box whose mail is broken.

The pieces are a small wrapper script and one edited crontab line. Installing the wrapper to /usr/local/bin needs root (sudo), and editing the root user's crontab needs sudo. If your job lives in an ordinary user's crontab, everything here runs unprivileged as that user. Nothing here deletes data.

Two things to treat with care:

  • The wrapper captures your job's stdout/stderr and sends it to the monitor instead of letting cron email it. If you currently rely on cron's MAILTO output, you are changing where that output goes. Read the script before you install it, and test it against a throwaway command first — not your production backup job.
  • Test on a non-production crontab or a test check. Create a test monitor, wire up a sleep 1; exit 0 job, watch it go green, then deliberately make it exit 1 and confirm you actually get paged. Only then point it at the real job.

To undo, you remove the wrapper from the crontab line and (optionally) delete the check. That is covered at the end.

Assumed environment: Debian 12 / Ubuntu 22.04, bash, curl installed (sudo apt install curl). I use Healthchecks as the monitor — it is open source, has a hosted free tier, and self-hosts cleanly. The same wrapper pattern works against any monitor that gives you a "ping this URL on success" endpoint.

Why plain cron isn't enough

Cron only emails you when a job produces output and a working MTA is configured and MAILTO is set. It tells you nothing when:

  • the job never fired (machine down, cron daemon dead, the line got commented out),
  • the job exited non-zero but printed nothing,
  • local mail is silently blackholed.

A dead man's switch inverts the logic. Instead of waiting for a failure message that may never arrive, you expect a regular success heartbeat. When the heartbeat stops, that absence is the alert.

Step 1 — Create the check

In the Healthchecks dashboard, create a new check. Set its Period to how often the job should run and its Grace to how long late is acceptable — a job that runs every 15 minutes might use a 5-minute grace. Set the Period at least as long as your job's worst-case runtime so a slow-but-successful run doesn't page you.

Each check has a unique ping URL, like:

https://hc-ping.com/00000000-0000-0000-0000-000000000000

Copy yours. That UUID is the only per-job value you'll substitute below.

Then configure at least one Integration on the check (email, Pushover, PagerDuty, Slack, whatever you actually watch). Without an integration, the monitor knows the job failed but has no way to reach you.

Step 2 — Install the wrapper

Save this as /usr/local/bin/hc-run and make it executable. It runs any command you hand it, then reports that command's exit code to the monitor and attaches the output as the ping body — so when you do get paged, the last run's log is right there.

#!/usr/bin/env bash
# hc-run — run a command and report success/failure to Healthchecks.
# Usage: hc-run <ping-url> <command> [args...]
set -uo pipefail   # NOT -e: we must not bail before we report the exit code

PING_URL="$1"; shift   # first argument is the check's ping URL

# Tell the monitor the job started (lets it measure run duration).
curl -fsS -m 10 --retry 5 -o /dev/null "${PING_URL}/start" || true

# Run the real job, capturing stdout+stderr together.
OUTPUT="$("$@" 2>&1)"
CODE=$?                 # exit code of the wrapped command

# Report the exit code (0 = up, non-zero = down) with the output as the body.
curl -fsS -m 10 --retry 5 -o /dev/null \
  --data-raw "$OUTPUT" \
  "${PING_URL}/${CODE}" || true

exit "$CODE"           # preserve the original exit code for cron
sudo install -m 0755 /dev/stdin /usr/local/bin/hc-run   # or: sudo chmod 755 after saving

Notes on the non-obvious lines:

  • set -uo pipefail without -e is deliberate. If -e were on, a failing job would abort the script before it reported the failure — exactly backwards.
  • The || true on the curl calls means a transient network blip talking to the monitor never changes the job's own exit status.
  • Appending /start, /${CODE}, or /fail to the ping URL is Healthchecks' documented convention. The /start and exit-status endpoints are described under "Measuring script run time" and "Reporting exit status" in the Healthchecks docs — confirm the exact paths there for your version, hosted or self-hosted.
  • --retry 5 and -m 10 give the ping a fair chance without hanging the job forever.

Step 3 — Wire it into the crontab

Edit the crontab that owns the job (crontab -e for a user, sudo crontab -e for root, or the relevant file in /etc/cron.d). Wrap the existing command with hc-run and its ping URL. Replace the UUID with your own and the command with your real job:

# Replace the UUID and the command with your own.
*/15 * * * * /usr/local/bin/hc-run https://hc-ping.com/00000000-0000-0000-0000-000000000000 /usr/local/bin/mybackup.sh

Keep the schedule you already had. hc-run is transparent — it runs your command and exits with your command's code, so anything downstream behaves as before.

For a /etc/cron.d file or a systemd timer's ExecStart, use the same full command string.

Step 4 — Verify it works

Do all three of these. The first two prove the plumbing; the third proves the silent case, which is the whole point.

1. A good run goes green. Run the wrapper by hand and watch the check flip to "up":

/usr/local/bin/hc-run https://hc-ping.com/00000000-0000-0000-0000-000000000000 /bin/true

2. A bad run pages you. Force a failure and confirm the alert lands and shows the output:

/usr/local/bin/hc-run https://hc-ping.com/00000000-0000-0000-0000-000000000000 \
  bash -c 'echo "pretend disk full"; exit 1'

You should get whatever integration you configured, with pretend disk full in the body.

3. The dead man's switch fires. In your test check, don't ping at all — just wait until Period + Grace elapses. The check should go "down" and alert on its own, with no failure message from the job, because the job never checked in. That absence-detection is what plain cron cannot do.

You can also confirm the live job actually runs through the wrapper by watching the check's "Last Ping" timestamp advance on schedule, or by tailing /var/log/syslog (Debian/Ubuntu) for the cron line — on RHEL/Alma that log is /var/log/cron.

Undo / rollback

This is fully reversible.

  • Remove the monitoring from one job: crontab -e (or sudo crontab -e) and strip the /usr/local/bin/hc-run https://... prefix, leaving the original command. Cron's default output behaviour returns immediately.
  • Remove the wrapper entirely: sudo rm /usr/local/bin/hc-run once no crontab references it.
  • Retire the check: delete or pause it in the Healthchecks dashboard so a decommissioned job doesn't page you forever.

If you'd rather self-host

Healthchecks ships an official Docker image and a Django app you can run behind your own reverse proxy; the ping URL then points at your own hostname instead of hc-ping.com, and everything above is unchanged. I won't walk the install here — see the project's self-hosting documentation for the current compose file and required environment variables.

For systemd-managed jobs specifically, there is a second mainstream approach worth knowing by name: a systemd timer with an OnFailure= unit that triggers a notifier. It handles the "job failed" case natively but not the "host is dead" case, which is why I reach for a dead man's switch first.

Written by
Ketan Aagja

Runs enterprise networks and security for a living, and writes Shore Up to turn two decades of hands-on Linux, Windows and mail-server work into guides you can actually use.

More about the author →

Was this article helpful?

Tap a star — no sign-in needed.

Be the first to rate this article.