Automatically Restart a Hung Service on a Schedule, with Logging

Some services don't crash cleanly. The process stays alive, systemctl still calls it active, but it has stopped answering — a wedged worker pool, a deadlocked thread, a leaked connection table. systemd's own Restart=on-failure handles a process that exits, but it won't help you with one that's technically running and doing nothing. This guide sets up a periodic health check that catches that case, restarts the unit when it's genuinely stuck, and logs every decision so you can prove what happened at 3 a.m.

Before you run this

What this does: it installs a small bash script plus a systemd timer that runs the script on a fixed interval (every 5 minutes in the example). The script checks whether your service is active and responding; if it isn't, it runs systemctl restart on the unit and writes a line to the system journal. It does not modify the service itself or its config.

Privileges: the watchdog script must run as root, because restarting a system-level service requires it. The systemd service unit that runs the script therefore runs as root by default. Editing files under /etc/systemd/system/ and running systemctl daemon-reload also require root — use sudo.

The one destructive thing here is the restart. A restart interrupts the service and drops its in-flight connections. That is the whole point, but it means a badly written health check (one that reports "unhealthy" when the service is fine) will restart a working service on a loop. Read the script, and test the health check by hand first on a non-production box or a test VM — run it, watch what it returns when the service is up and when you deliberately stop the service — before you enable the timer anywhere real.

This is reversible. Disabling the timer and removing the unit files returns the box to exactly where it started; the undo steps are at the end.

Assumptions

I'm writing for Debian 12 / Ubuntu 22.04 with systemd. On RHEL/Alma the paths and commands are identical; only the package names for anything you install (like curl) differ. I assume:

• The unit you want to watch is a system service — I'll call it myapp.service. Replace that everywhere.
• You have a cheap way to tell whether the service is responding, not just running. In the example that's an HTTP health endpoint. If your service isn't HTTP, you'll swap that one check for something appropriate (a port probe, a query, a status command) — I'll flag exactly where.

The health-check script

Save this as /usr/local/sbin/service-watchdog.sh:

#!/usr/bin/env bash
set -euo pipefail

SERVICE="myapp.service"                    # <-- replace with your unit name
HEALTH_URL="http://127.0.0.1:8080/health"  # <-- replace, or swap the check below
LOG_TAG="service-watchdog"

log() {
    # logger writes to the journal/syslog with a tag we can filter on later
    logger -t "$LOG_TAG" "$1"
}

# 1. Is the unit even active? If not, restart and stop here.
if ! systemctl is-active --quiet "$SERVICE"; then
    log "WARNING: $SERVICE is not active; restarting."
    systemctl restart "$SERVICE"
    exit 0
fi

# 2. Unit is active, but is it actually responding?
#    Replace this curl with whatever proves your service is healthy.
if curl --fail --silent --max-time 5 "$HEALTH_URL" >/dev/null; then
    log "OK: $SERVICE is healthy."
else
    log "WARNING: $SERVICE is active but the health check failed; restarting."
    systemctl restart "$SERVICE"
fi

Make it executable and root-owned:

sudo chown root:root /usr/local/sbin/service-watchdog.sh
sudo chmod 750 /usr/local/sbin/service-watchdog.sh

The check in step 2 is the part you must adapt. curl --fail returns non-zero on an HTTP error status or a connection failure, and --max-time 5 means a hung service that accepts the connection but never answers will time out and count as unhealthy — which is exactly the wedged case we're after. If your service isn't HTTP, replace that curl line with a command that returns 0 when healthy and non-zero when not: a ss -H -ltn 'sport = :5432' port check, a pg_isready, an application-specific status command. Keep the timeout — a health check with no timeout can itself hang.

Test it by hand before going further:

sudo /usr/local/sbin/service-watchdog.sh

Then stop the service (sudo systemctl stop myapp.service) and run it again — it should restart the service and you should see the warning in the journal (checked below). Only once both paths behave should you schedule it.

The systemd service and timer

Rather than cron, I'll use a systemd timer. It logs to the journal for free, survives reboots cleanly, and its state is visible with systemctl list-timers. (cron works fine for this too; I'm just picking one path and holding to it.)

Create /etc/systemd/system/service-watchdog.service:

[Unit]
Description=Health check and conditional restart for myapp

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/service-watchdog.sh

Type=oneshot is correct here: the script runs, does its work, and exits — this isn't a long-running daemon.

Create /etc/systemd/system/service-watchdog.timer:

[Unit]
Description=Run the myapp watchdog periodically

[Timer]
OnBootSec=2min          # first run 2 minutes after boot
OnUnitActiveSec=5min    # then every 5 minutes after the last run
Persistent=true

[Install]
WantedBy=timers.target

Adjust OnUnitActiveSec to how often you want to check. Don't set it so tight that a legitimately slow startup gets killed before it finishes — leave more time than your service needs to boot.

Load it and enable the timer (not the service — the timer triggers the service):

sudo systemctl daemon-reload
sudo systemctl enable --now service-watchdog.timer

Verify it's working

Confirm the timer is registered and see when it next fires:

systemctl list-timers service-watchdog.timer

Trigger a run immediately without waiting for the schedule:

sudo systemctl start service-watchdog.service

Read what the script logged. Because we tagged it with logger -t service-watchdog, filter on that tag:

journalctl -t service-watchdog --since "10 min ago"

You should see OK: lines when the service is healthy. To prove the restart path, stop the service and run the watchdog service unit once more, then check the journal again — you should see the WARNING ... restarting line, and systemctl is-active myapp.service should report active again.

If you'd rather also keep a plain-text file, journald is already the durable record; you can export it on demand with journalctl -t service-watchdog > /path/to/watchdog.log. I'd avoid having the script write its own file in parallel — one source of truth is easier to trust.

A note on doing less

If your only problem is a service that crashes (exits non-zero), you don't need any of this — set Restart=on-failure and a RestartSec= in the unit's [Service] section and let systemd handle it. And if your application implements systemd's watchdog protocol (sd_notify), systemd can restart it on a missed heartbeat via WatchdogSec= — see the systemd.service man page (man systemd.service) for the exact directives before relying on it. The script in this guide exists specifically for the case where the process is alive but unresponsive and doesn't speak that protocol.

Undo

To stop and remove everything:

sudo systemctl disable --now service-watchdog.timer
sudo rm /etc/systemd/system/service-watchdog.timer
sudo rm /etc/systemd/system/service-watchdog.service
sudo rm /usr/local/sbin/service-watchdog.sh
sudo systemctl daemon-reload

daemon-reload clears systemd's memory of the removed units. The watched service itself is untouched by any of this — you've only removed the thing that was watching it.