Auto-Restart Failing systemd Services with Backoff

Before you run this

This guide configures systemd to automatically restart a service when it fails, with an increasing delay between attempts (backoff), and to fire an alert when it exhausts its retries and stays down. Nothing here touches your application's data — it only changes how systemd supervises the unit — but a badly tuned restart loop can hammer CPU, disk, or an upstream database by relaunching a broken process over and over, so treat the values as production settings, not guesses.

You need root / sudo to edit unit files under /etc/systemd/system and to run systemctl daemon-reload. The changes are made through a drop-in override, which is fully reversible with systemctl revert — no original file is overwritten.

Read the whole thing first, then try it on a throwaway service on a test VM, not on your production database or web server. I include a deliberately-failing scratch unit below exactly so you can watch the backoff behave before you touch anything real. Confirm your systemd version first, because the backoff directives I rely on only exist in newer releases.

Assumptions

I'm writing for Debian 12 / Ubuntu 22.04+ with systemd, editing units under /etc/systemd/system. The concepts are identical on RHEL/Alma; only the base OS differs. I assume the service you want to protect already exists as a unit called myapp.service — substitute your real unit name everywhere you see it.

Check your systemd version first

The growing-delay directives (RestartSteps=, RestartMaxDelaySec=) were introduced in systemd v254. Everything else works on older releases.

systemctl --version | head -1

If that number is 254 or higher (Ubuntu 24.04, Debian 13, and current Fedora qualify), you get true backoff. If it's lower (Ubuntu 22.04 ships 249, RHEL 9 ships 252), skip the backoff directives and use the fixed-delay version I note below — systemd on those releases has no native exponential backoff, and I'd rather you use the stable fixed delay than reach for an unofficial wrapper.

The building blocks

Three directives do the core work, and one common mistake trips people up: the rate-limit keys live in [Unit], not [Service].

• Restart=on-failure — restart only when the service exits non-zero or is killed by a signal, not on a clean exit 0.
• RestartSec= — how long to wait before the first restart.
• StartLimitIntervalSec= and StartLimitBurst= — if the service fails Burst times within Interval, systemd stops trying and puts the unit into the failed state. This is what stops an infinite crash loop.

Because a service being auto-restarted is not yet "failed," the unit only reaches the failed state once it blows past the start limit. That's the moment we want to be alerted — it means the automatic recovery gave up.

Add the backoff (systemd 254+)

RestartSteps= tells systemd how many increasing steps to take between RestartSec (the starting delay) and RestartMaxDelaySec (the ceiling). All three must be set for the ramp to work.

Create the override interactively — systemctl edit writes a drop-in and reloads for you:

sudo systemctl edit myapp.service

In the editor, add:

[Unit]
# Give up if it fails 5 times within 10 minutes
StartLimitIntervalSec=600
StartLimitBurst=5

[Service]
Restart=on-failure
RestartSec=1s              # first retry after 1 second
RestartSteps=10            # ramp the delay over 10 steps
RestartMaxDelaySec=5min    # cap the delay at 5 minutes

Save and exit. systemctl edit runs the daemon reload automatically; if you write the file by hand instead, run sudo systemctl daemon-reload yourself.

On systemd older than 254, drop the three backoff-specific lines and use a single fixed delay:

[Service]
Restart=on-failure
RestartSec=30s

Alert me when it finally gives up

OnFailure= activates another unit when this one enters the failed state. I point it at a template unit (@) so a single notifier serves every protected service, with %n passing the failed unit's name through.

Add to the same override:

[Unit]
OnFailure=notify-failure@%n.service

Now create the template notifier at /etc/systemd/system/notify-failure@.service:

[Unit]
Description=Alert that %i entered failed state

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure.sh %i

And the script it runs, at /usr/local/bin/notify-failure.sh:

#!/usr/bin/env bash
set -euo pipefail

unit="$1"

# logger is part of util-linux and is always present.
logger -t notify-failure "systemd unit ${unit} entered failed state on $(hostname)"

# Add your own channel below — e.g. an SMTP relay via 'mail',
# or a webhook via curl. Check that tool's own docs for exact
# flags before pasting into production; I'm not going to guess
# your mailer's syntax here.

Make it executable and reload:

sudo chmod +x /usr/local/bin/notify-failure.sh
sudo systemctl daemon-reload

I deliberately keep the notifier to logger (guaranteed present) and leave the mail/webhook line as prose. If you wire in mail, sendmail, or a Slack/Teams webhook with curl, verify that tool's own arguments against its documentation rather than trusting a snippet.

Test it on a scratch unit first

Before you touch myapp.service, prove the behaviour with a unit that fails on purpose. Create /etc/systemd/system/backoff-test.service:

[Unit]
Description=Backoff test - fails every time

[Service]
ExecStart=/bin/sh -c 'echo starting; sleep 2; exit 1'
Restart=on-failure
RestartSec=1s
RestartSteps=5
RestartMaxDelaySec=30s

Then watch the delays grow:

sudo systemctl daemon-reload
sudo systemctl start backoff-test.service
journalctl -u backoff-test.service -f

You'll see the gap between "starting" lines widen — 1s, then longer, up toward 30s — until systemd hits the default start limit and marks the unit failed. When you're satisfied, remove the scratch unit:

sudo systemctl stop backoff-test.service
sudo rm /etc/systemd/system/backoff-test.service
sudo systemctl daemon-reload

Verify the real service

Confirm the drop-in merged correctly — this prints the effective unit with your override appended:

systemctl cat myapp.service

Check the values systemd actually parsed:

systemctl show myapp.service | grep -iE 'restart|startlimit'

Confirm the notifier is armed:

systemctl show myapp.service -p OnFailure

To exercise the alert path safely, run the notifier by hand rather than crashing production:

sudo /usr/local/bin/notify-failure.sh myapp.service
journalctl -t notify-failure -n 5

You should see your test line in the journal. If you later want to see the full chain fire for real, do it in a maintenance window against the scratch unit above, not the live service.

Undo

The override is a drop-in, so reverting is clean — this deletes the drop-in and reloads:

sudo systemctl revert myapp.service

Remove the notifier if you no longer want it:

sudo rm /etc/systemd/system/notify-failure@.service
sudo rm /usr/local/bin/notify-failure.sh
sudo systemctl daemon-reload

If a unit is stuck in the failed state after testing, clear the counter with systemctl reset-failed myapp.service so it will start and count fresh again.

For the exact semantics of each directive — especially how RestartSteps interpolates between RestartSec and RestartMaxDelaySec, and how the start-limit counter interacts with Restart= — see the systemd.service and systemd.unit man pages, which are the authoritative reference and match whatever version you actually have installed (man systemd.service).