Monitor systemd Socket Activation and Alert on Queue Depth

When systemd hands a listening socket to a service, the kernel keeps completed connections in an accept queue until the service calls accept() on them. If the service falls behind — thread-starved, blocked on a slow database, whatever — that queue fills. Past the socket's backlog it silently drops or refuses connections. This guide sets up a small, read-only monitor that watches the accept-queue depth on your listening sockets and raises an alert before you hit that wall.

Before you run this

What it does: the script below reads the current accept-queue depth (Recv-Q) of your listening TCP sockets using ss, and writes a warning to the journal (optionally emails you) when any socket's queue exceeds a threshold you set. A systemd timer runs it on a schedule. That's the whole job.

It changes nothing on your running services. It only reads socket statistics and writes log lines. It does not touch your firewall, your socket units, or any connection. There is nothing to corrupt and nothing irreversible here — the only "undo" is removing the timer and the script, which I cover at the end.

Privileges: reading listening-socket queue depths with ss -ltn works fine as an unprivileged user, because we do not ask for process information (-p, which would need root). You do need root to write the script into /usr/local/sbin and to install the systemd unit files. Installing the timer to run as root is the normal choice so it can see every socket.

Test first: read the script before you run it. Try it on a test VM or a non-production host, and prove the alert fires by temporarily setting the threshold very low (THRESHOLD=0) so you can see a journal line appear, then set it back to a sane value. Don't deploy the timer straight onto a busy production box without watching it run by hand once.

Assumptions: Ubuntu 22.04 LTS, systemd 249, bash 5, and iproute2 recent enough to support ss --no-header (-H). On RHEL/Alma the commands are identical; only package names differ (logger ships in util-linux, mail sends via s-nail/mailx rather than mailutils). If your ss is old enough that -H errors out, replace ss -ltnH with ss -ltn | tail -n +2.

Understand what you're measuring

For a socket in LISTEN state, ss reports two useful numbers:

• Recv-Q — the number of established connections sitting in the accept queue, waiting for the service to accept them.
• Send-Q — the maximum size of that queue, i.e. the socket's backlog.

That backlog is exactly the Backlog= value on the .socket unit (default SOMAXCONN when unset — see man systemd.socket). So a healthy socket-activated service keeps Recv-Q near zero; a Recv-Q creeping toward Send-Q is your warning that the service can't keep up.

First, see which sockets systemd is actually managing:

# List every socket unit systemd is listening on, and what each activates
systemctl list-sockets

Then look at the live queue depths:

# -l listening, -t TCP, -n numeric (no DNS/port name lookups)
ss -ltn

State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port  Process
LISTEN  0       128     0.0.0.0:22          0.0.0.0:*
LISTEN  17      100     0.0.0.0:80          0.0.0.0:*

That second line — 17 connections queued against a backlog of 100 — is the kind of thing worth alerting on before it reaches 100.

The monitor script

Save this as /usr/local/sbin/socket-queue-monitor.sh and chmod 750 it. Substitute your own values for the obvious placeholders (alerts@example.com, and the ports in WATCH_PORTS if you use it).

#!/usr/bin/env bash
set -euo pipefail

# Alert when a listening socket's accept queue (Recv-Q) crosses a threshold.
# Recv-Q on a LISTEN socket = connections that finished the TCP handshake
# but the service has not yet accept()ed. A rising value means it's behind.

THRESHOLD="${THRESHOLD:-50}"          # queue depth that triggers an alert
WATCH_PORTS="${WATCH_PORTS:-}"        # optional: space-separated ports; empty = all

alert() {
    local recvq=$1 sendq=$2 addr=$3
    # Log to the journal at daemon.warning so journalctl and any log forwarder see it
    logger -t socket-queue-monitor -p daemon.warning \
        "accept queue ${recvq}/${sendq} on ${addr} over threshold ${THRESHOLD}"

    # Optional email — uncomment if this host has a working MTA / mail command:
    # printf 'accept queue %s/%s on %s over %s\n' "$recvq" "$sendq" "$addr" "$THRESHOLD" \
    #   | mail -s "socket queue alert on $(hostname -s)" alerts@example.com
}

# ss -H suppresses the header; fields are: state recvq sendq local peer
ss -ltnH | while read -r state recvq sendq addr peer _; do
    # If a port allow-list is set, skip sockets not on it
    if [[ -n "$WATCH_PORTS" ]]; then
        port="${addr##*:}"                     # port = text after the last colon
        [[ " $WATCH_PORTS " == *" $port "* ]] || continue
    fi
    if (( recvq > THRESHOLD )); then
        alert "$recvq" "$sendq" "$addr"
    fi
done

By default it checks every listening TCP socket. To narrow it to the ports your socket-activated services actually use — read them off systemctl list-sockets — set WATCH_PORTS="80 443" in the service unit below.

If your socket-activated service listens on a Unix domain socket rather than TCP, the same Recv-Q/Send-Q idea applies; list those with ss -lxH instead of ss -ltnH. I'm keeping this guide to TCP because that's the common alerting case; adapt the ss invocation if you need Unix sockets too.

Run it on a timer

Create /etc/systemd/system/socket-queue-monitor.service:

[Unit]
Description=Alert when listening socket accept queues exceed threshold

[Service]
Type=oneshot
Environment=THRESHOLD=50
# Environment=WATCH_PORTS=80 443
ExecStart=/usr/local/sbin/socket-queue-monitor.sh

And /etc/systemd/system/socket-queue-monitor.timer:

[Unit]
Description=Run the socket queue monitor every minute

[Timer]
OnBootSec=2min
OnUnitActiveSec=1min

[Install]
WantedBy=timers.target

Load and enable:

systemctl daemon-reload
systemctl enable --now socket-queue-monitor.timer

A one-minute interval is a sensible default for a queue that can fill in seconds under load. If that's too chatty, raise OnUnitActiveSec. Note that a timer samples — a queue spike lasting a few seconds between runs can be missed. If you need continuous visibility, feed the same ss numbers into your existing metrics system (a node_exporter textfile collector, say) rather than a timer; that's a different article, but the source of truth is the same ss command.

Verify it works

Run the script by hand first and watch the journal:

# Force it to alert on everything so you can see output, then run once
sudo THRESHOLD=0 /usr/local/sbin/socket-queue-monitor.sh
journalctl -t socket-queue-monitor --no-pager -n 20

You should see one warning line per listening socket. Set THRESHOLD back to a real value before deploying.

Confirm the timer is scheduled and the service ran cleanly:

systemctl list-timers socket-queue-monitor.timer
systemctl status socket-queue-monitor.service
journalctl -t socket-queue-monitor --since "10 min ago"

list-timers shows the next and last trigger times; status on the oneshot service shows the last exit (it should be status=0/SUCCESS after each run).

Undo it

Nothing here persists beyond the files you created, and no service was modified. To remove it completely:

sudo systemctl disable --now socket-queue-monitor.timer
sudo rm /etc/systemd/system/socket-queue-monitor.timer
sudo rm /etc/systemd/system/socket-queue-monitor.service
sudo rm /usr/local/sbin/socket-queue-monitor.sh
sudo systemctl daemon-reload

Your journal will still hold whatever warnings were logged; those age out with normal journal rotation and need no cleanup.

For the underlying tools, man ss documents the Recv-Q/Send-Q semantics for listening sockets, and man systemd.socket covers Backlog= and the rest of the socket-unit options if you decide to raise a backlog rather than just alert on it.