Automate Failover Testing Between Primary and Secondary Mail Servers

Having two MX hosts is not the same as knowing failover works. The only way to trust it is to test it: confirm both hosts answer on port 25 and accept mail for your domain, and once in a while prove that when the primary is down, the secondary still takes the message and holds it. This guide gives you a small probe script you can run on a schedule, and a controlled live drill for the real thing.

Before you run this

The probe script (mx-failover-check.sh below) is non-destructive. For each MX host of your domain it opens an SMTP session, says HELO, and issues MAIL FROM / RCPT TO — then quits before sending any message body. Nothing is delivered, no mailbox is touched. It only tells you whether each host is reachable and willing to accept mail for a recipient you name.

• Privileges: the probe needs no root. It runs as any user who can reach TCP 25 outbound. It reads DNS and talks SMTP; it changes nothing.
• Test first: read the script, then run it once by hand and watch the log before you put it in cron. Point PROBE_TO at an address your servers already accept (postmaster is the safe default) so you are not creating bounces.
• The live drill is different. Actually stopping the primary MTA to watch mail roll to the secondary does change a running system: while the primary is stopped it accepts no mail, and existing sessions are cut. Do it only in a maintenance window. It is fully reversible — starting the service again restores service — but treat it as a real outage while it lasts.

Assumed environment: Debian 12 / Ubuntu 22.04, bash, Postfix on both the primary and the backup MX, with the backup already configured as a relay/backup MX for your domain. On RHEL/Alma the package and service names differ where noted; the logic is identical.

Install the tools

The probe uses swaks (a standard command-line SMTP test client) and dig.

# Debian / Ubuntu
sudo apt install swaks dnsutils

# RHEL / Alma (swaks is in EPEL; dig comes from bind-utils)
# sudo dnf install epel-release && sudo dnf install swaks bind-utils

The probe script

Save this as mx-failover-check.sh. Replace the three placeholder values at the top — the domain, and a sender/recipient your servers accept.

#!/usr/bin/env bash
#
# mx-failover-check.sh — probe every MX host for a domain and confirm each
# one is reachable and accepts mail for a known recipient.
# Non-destructive: the SMTP conversation stops after RCPT TO; no body is sent.

set -euo pipefail

DOMAIN="example.com"                    # domain whose MX records you are testing
PROBE_FROM="failover-test@example.com"  # envelope sender for the probe
PROBE_TO="postmaster@example.com"       # recipient every MX for the domain should accept
PORT=25
HELO="$(hostname -f)"                   # HELO name; some servers reject a bare hostname
LOG="/tmp/mx-failover-$(date +%Y%m%d-%H%M%S).log"

# MX hosts, lowest preference number first (primary first), trailing dot stripped.
mapfile -t MX < <(dig +short MX "$DOMAIN" | sort -n | awk '{print $2}' | sed 's/\.$//')

if [ "${#MX[@]}" -eq 0 ]; then
  echo "No MX records found for $DOMAIN" >&2
  exit 1
fi

echo "Testing ${#MX[@]} MX host(s) for $DOMAIN — full transcript: $LOG"
fail=0

for host in "${MX[@]}"; do
  printf '=== %s ===\n' "$host" >>"$LOG"
  # --quit-after RCPT stops after the recipient is accepted/rejected, so no mail is sent.
  if swaks --server "$host" --port "$PORT" \
           --helo "$HELO" \
           --from "$PROBE_FROM" \
           --to "$PROBE_TO" \
           --quit-after RCPT >>"$LOG" 2>&1; then
    echo "OK   $host accepted RCPT for $PROBE_TO"
  else
    echo "FAIL $host did not accept the probe (see $LOG)"
    fail=1
  fi
done

exit "$fail"

Make it executable and run it:

chmod +x mx-failover-check.sh
./mx-failover-check.sh

Expected output when both MX hosts are healthy:

Testing 2 MX host(s) for example.com — full transcript: /tmp/mx-failover-20240101-120000.log
OK   mx1.example.com accepted RCPT for postmaster@example.com
OK   mx2.example.com accepted RCPT for postmaster@example.com

The script exits 0 when every host passes and 1 if any fails, which is what makes it usable from a scheduler or a monitoring check.

The --quit-after value and the --server / --from / --to flags are all standard swaks options; if you want to extend the probe (for example to force STARTTLS or test authentication), check man swaks for the exact flag names rather than guessing — swaks has a lot of options and they are precise.

Run it on a schedule

A simple cron entry that mails you only when something fails:

# Probe MX failover every 15 minutes; email root only on non-zero exit.
*/15 * * * * /usr/local/bin/mx-failover-check.sh >/dev/null || echo "MX failover probe FAILED for example.com" | mail -s "MX probe alert" root

That relies on cron's own mailing plus a working local mailer. If you already run Prometheus, Nagios, or Zabbix, call the script from a check and let the monitoring system own alerting instead — the exit code is designed for exactly that.

The live failover drill

The probe proves reachability. It does not prove that mail survives a real primary outage — for that you must take the primary down and watch a message land on and stay queued at the secondary. Do this in a maintenance window.

On the primary MTA:

# Stop only the mail service. Existing SMTP sessions drop; the host stays up.
sudo systemctl stop postfix

(On RHEL/Alma the unit is also postfix. If you run a different MTA, stop that unit instead.)

Now, from a third machine, send a real test message addressed to your domain and let DNS/MX selection route it. Because the primary refuses connections, the sender should fall through to the secondary:

swaks --to postmaster@example.com --from you@another-domain.example \
      --header "Subject: failover drill $(date -Iseconds)"

Without --server, swaks resolves the domain's MX records and connects to them in preference order — so this exercises the same selection logic a real sender uses.

On the secondary MX, confirm it accepted the message and is holding it for the primary:

# List the queue; you should see the drill message deferred, waiting for the primary.
postqueue -p
# Watch the log as it accepts the message and defers delivery to the down primary.
sudo tail -f /var/log/mail.log     # RHEL/Alma: /var/log/maillog

You should see the secondary accept the message, then defer delivery to the primary with a connection error — exactly the correct behaviour for a backup MX.

Verify and roll back

Bring the primary back:

sudo systemctl start postfix
systemctl status postfix          # confirm active (running)

Restarting the service is the whole undo — there is nothing else to reverse. Once the primary is up, force the secondary to flush its queue so the held message is delivered rather than waiting for the retry timer:

# On the secondary MX:
sudo postqueue -f
postqueue -p                      # should now show "Mail queue is empty"

Finally, run the probe once more to confirm both hosts are green again:

./mx-failover-check.sh

Two closing sanity checks worth doing after any drill. First, confirm the secondary actually relays for your domain rather than just accepting and bouncing — that depends on your relay_domains and recipient-verification settings, which are Postfix configuration on the backup host; see the Postfix documentation for relay_domains and relay_recipient_maps. A backup MX that accepts mail but has no valid recipient list becomes a backscatter source. Second, confirm the delivered drill message reached the destination mailbox, so you know the full path — sender → secondary → primary → mailbox — worked end to end, not just the first hop.