Alert Before a Runaway Log File Fills Your Disk

A full disk is one of those failures that takes half your services down at once — the mail queue stops, the database goes read-only, journald starts dropping entries — and it almost always announces itself in advance as a slowly climbing df percentage. This guide sets up a small, boring watchdog that checks free space and the biggest offenders under your log directories, then emails you before the disk is full instead of after.

Before you run this

What it does: the script below is read-only. It runs df, find, and du to report disk usage and list the largest files under a directory you choose, and it sends you an email when a filesystem crosses a usage threshold. It does not delete, rotate, truncate, or move a single byte. That is deliberate — an automated cleaner that guesses wrong is far more dangerous than a full disk, so this tool only tells you.

Privileges: to see every file under /var/log you generally need root, because some logs are mode 0600 owned by services. Run the manual tests with sudo, and install the cron job under root (or a user in the relevant groups) so the scan isn't silently missing files it can't read. The df and threshold logic work fine unprivileged; only the per-file du/find listing needs the extra reach.

Test first: read the script before you install it. Run it by hand once with a deliberately low threshold so you can confirm the alert path works, then raise the threshold to something real. Do this on a test VM or a non-production host first if you're wiring it to a mail relay you haven't used from this box before — the most common failure here is that mail silently never leaves the machine.

Assumptions: Debian 12 / Ubuntu 22.04, bash, GNU coreutils (df, du), GNU findutils, and a working local mail path via the mail command from the mailutils package. On RHEL/Alma the equivalent package is mailx and the command is often mailx; the df --output= and find -size syntax below is GNU-specific and is the same on both, but confirm your mail/mailx invocation. I use cron here because it's universal; a systemd timer is the other mainstream option and I note it at the end.

This changes nothing on disk, so there is nothing to roll back except the cron entry itself, which I cover under Undo.

Install the mail tool

If you don't already have a way to send mail from the shell:

sudo apt update
sudo apt install mailutils          # provides the `mail` command

mailutils will pull in an MTA. If this host already relays through Postfix or an external smarthost, keep that; don't let the installer reconfigure a working mail setup. Confirm mail actually leaves the box:

echo "test from $(hostname)" | mail -s "mail test" you@example.com

If that message doesn't arrive, fix mail delivery before going any further — the watchdog is useless if its alerts don't reach you.

The script

Save this as /usr/local/sbin/disk-log-watch.sh. Replace the three placeholders at the top: the email address, the threshold percentage, and the directory you want the "largest files" report to cover.

#!/usr/bin/env bash
set -euo pipefail

# ---- settings you edit ----
ALERT_EMAIL="you@example.com"     # where alerts go
THRESHOLD=85                      # alert when a filesystem is at/above this % used
WATCH_DIR="/var/log"              # directory to list biggest files from
BIG_FILE_MB=100                   # in the report, flag files larger than this (MiB)
# ---------------------------

hostname="$(hostname -f 2>/dev/null || hostname)"
alert_needed=0
report=""

# df --output is GNU coreutils; pcent and target are valid field names.
# We skip the header with tail, and read percent + mountpoint per line.
while read -r pcent target; do
    # strip the trailing % and any whitespace
    used="${pcent%\%}"
    used="${used// /}"
    [ -z "$used" ] && continue
    if [ "$used" -ge "$THRESHOLD" ]; then
        alert_needed=1
        report+="FILESYSTEM ${target} is at ${used}% used"$'\n'
    fi
done < <(df --output=pcent,target -x tmpfs -x devtmpfs | tail -n +2)

if [ "$alert_needed" -eq 1 ]; then
    report+=$'\n'"Largest files under ${WATCH_DIR} (over ${BIG_FILE_MB} MiB):"$'\n'
    # -size +NM is MiB in GNU find; -printf gives size in bytes + path.
    # Sort numerically, largest first, take the top 20.
    report+="$(find "$WATCH_DIR" -type f -size "+${BIG_FILE_MB}M" -printf '%s\t%p\n' 2>/dev/null \
        | sort -rn \
        | head -n 20 \
        | awk -F'\t' '{ printf "%8.1f MiB  %s\n", $1/1048576, $2 }')"

    subject="[disk alert] ${hostname}: filesystem over ${THRESHOLD}%"
    printf '%s\n' "$report" | mail -s "$subject" "$ALERT_EMAIL"
fi

A few notes on the non-obvious lines:

• df --output=pcent,target limits df to just the percentage and mount point, which is far easier to parse than the default columns. -x tmpfs -x devtmpfs excludes RAM-backed filesystems that always sit near 100% and would spam you.
• find ... -size +100M uses M for mebibytes in GNU find; the %s\t%p in -printf prints byte size and path, which I convert to MiB in the awk line so the report is readable.
• set -euo pipefail makes the script fail loudly rather than half-run. The find errors are suppressed with 2>/dev/null only because unreadable files under /var/log are expected noise when run as a non-root user — run it as root and you'll see everything.

Make it executable:

sudo chmod 0750 /usr/local/sbin/disk-log-watch.sh

Schedule it

Install it in root's crontab so it runs every 15 minutes:

sudo crontab -e

Add this line:

*/15 * * * * /usr/local/sbin/disk-log-watch.sh

Cron mails root any output a job prints; this script prints nothing unless something is genuinely broken (set -e firing), so a message from cron itself means the script errored — worth investigating. The disk alert itself comes from the mail command inside the script and goes to your ALERT_EMAIL.

Verify it works

Test the alert path before you trust it. Temporarily edit the script and set THRESHOLD=0, then run it by hand as root:

sudo /usr/local/sbin/disk-log-watch.sh

With the threshold at 0 every real filesystem is "over", so you should receive an email listing your mounts and the largest files under WATCH_DIR. Confirm the mail arrives and the file list looks sane. Then set THRESHOLD back to a real value (I use 85; on a small root filesystem 80 buys you more reaction time).

Confirm the schedule is registered:

sudo crontab -l | grep disk-log-watch

You can also compare the script's view against reality any time:

df -h                                   # what percentage each mount is at
sudo du -ah /var/log | sort -rh | head  # biggest log files, largest first

Undo

There is nothing on disk to reverse — the script only reads. To stop it, remove the cron line:

sudo crontab -e        # delete the disk-log-watch.sh line, save

And if you want it gone entirely:

sudo rm /usr/local/sbin/disk-log-watch.sh

If you prefer a systemd timer

The other mainstream way to schedule this is a systemd service unit plus a .timer unit with OnCalendar=*:0/15. It gives you systemctl status and journal logging for free. The script is identical; only the scheduling wrapper changes. See the systemd.timer man page for the exact unit syntax rather than copying a half-remembered one — the field names matter.

The natural next step once this is alerting reliably is to fix the cause: a service logging too verbosely, or logrotate not rotating something because it lives outside /var/log. But get the warning working first. Knowing the disk is at 85% at 3 a.m. is the whole point.