Shore Up
A cluster of small houses each sending a paper note along a string to one large post office building, where a clerk files each note into a labelled drawer for that house.
LinuxSecurity

Ship Logs to a Central Syslog Server with rsyslog

Ketan Aagja9 min read
No ratings yet

Before you run this

This guide configures rsyslog on your Linux hosts to forward their system logs over the network to one central collector, and configures that collector to receive them and file each sender's logs into its own directory. The point is to have every machine's logs in one place so you can search, retain, and back them up centrally.

  • Privileges: You need sudo/root on both the collector and every client. rsyslog runs as root, its config lives under /etc/, and it binds a privileged port (514). Editing config and restarting the service both require root.
  • Test first: Read each config file before you deploy it. Try this on a test VM or a single non-production host before you touch fleet machines. rsyslog will refuse to start cleanly on a bad config and you can lose local logging on that host until you fix it, so validate (rsyslogd -N1) before every restart.
  • What it changes: You add new .conf files under /etc/rsyslog.d/ and restart the rsyslog service. This is fully reversible — delete the files you added and restart. Nothing is deleted or overwritten on existing hosts. On the collector you will create a new log directory and it will grow over time, so make sure that filesystem has room and is covered by log rotation.
  • Network note: Plain syslog over TCP/UDP is unencrypted. Only do this over a trusted network or a VPN. Encrypting syslog with TLS is a real, supported rsyslog feature but it is out of scope here; see the rsyslog documentation on TLS-encrypted syslog if you need it.

Assumptions

I'm assuming Debian 12 or Ubuntu 22.04/24.04 on both ends, with the stock rsyslog package installed (it is by default) and systemd managing it. I'll forward over TCP on port 514, which is the common, reliable choice. On RHEL/Alma the package and syntax are identical; the differences are the local log file path (/var/log/messages instead of /var/log/syslog), the firewall tool (firewalld instead of ufw), and SELinux, which I note where it matters.

I'll use modern rsyslog "advanced" (RainerScript) syntax throughout, which is standard on any rsyslog v8 shipped this decade.

Replace central.example.com and 10.0.0.10 below with your real collector's hostname and IP.

Step 1 — Configure the central collector to receive logs

On the collector, create a new file so you don't disturb the shipped config:

sudo tee /etc/rsyslog.d/10-remote-server.conf >/dev/null <<'EOF'
# Load the TCP input module and listen on 514
module(load="imtcp")

# Build a per-sender file path from the sender's IP and the program name
template(name="RemoteHostFile" type="string"
         string="/var/log/remote/%FROMHOST-IP%/%PROGRAMNAME%.log")

# A ruleset that writes anything it receives using that template
ruleset(name="remoteIntake") {
    action(type="omfile" dynaFile="RemoteHostFile")
}

# Bind the listener to the ruleset so remote logs are kept
# separate from this host's own local logs
input(type="imtcp" port="514" ruleset="remoteIntake")
EOF

Create the base directory (rsyslog creates the per-host subdirectories itself):

sudo mkdir -p /var/log/remote

Validate the config before restarting — this catches typos without stopping the running service:

sudo rsyslogd -N1

If it reports "End of config validation run. Bye.", restart:

sudo systemctl restart rsyslog

Open the firewall for TCP 514:

# Ubuntu / Debian with ufw
sudo ufw allow 514/tcp

# RHEL / Alma with firewalld
# sudo firewall-cmd --add-port=514/tcp --permanent && sudo firewall-cmd --reload

SELinux note (RHEL/Alma): port 514/tcp is already labelled syslogd_port_t, so the default listener works. If you ever change to a non-standard port you must relabel it with semanage port; check the rsyslog and SELinux docs for the exact command before doing that.

Confirm rsyslog is now listening:

sudo ss -tlnp | grep ':514'

You should see rsyslogd bound to 0.0.0.0:514.

Step 2 — Configure a client to forward its logs

On a client host, create a forwarding config. I'll use the omfwd action with a disk-assisted queue so that if the collector is unreachable, logs are buffered and retried instead of dropped:

sudo tee /etc/rsyslog.d/60-forward.conf >/dev/null <<'EOF'
# Forward all facilities/priorities to the central collector over TCP.
# The queue buffers messages if the collector is down and retries forever.
*.* action(type="omfwd"
           target="central.example.com"   # <-- your collector
           port="514"
           protocol="tcp"
           queue.type="linkedList"
           queue.filename="fwd_central"    # spool file name if it overflows to disk
           queue.saveonshutdown="on"       # keep queued logs across a restart
           action.resumeRetryCount="-1")   # retry indefinitely, never give up
EOF

Note that *.* forwards everything and the host also keeps writing its own local logs as before — forwarding is additional, not a replacement. If you only want, say, auth and above, you'd change the selector (e.g. authpriv.*); the selector syntax is the standard facility.priority form documented in the rsyslog manual.

Validate and restart:

sudo rsyslogd -N1
sudo systemctl restart rsyslog

Repeat Step 2 on each host you want to forward. This is a natural fit for pushing the single file out with Ansible's copy or template module plus a handler that restarts rsyslog, but the config itself is what matters.

Step 3 — Verify end to end

From a client, send a test message with a tag you'll recognise:

logger -t shoreup-test "forwarding check from $(hostname) at $(date)"

On the collector, that host should now have its own directory. Look for the message:

# Per-host directory, keyed by the client's IP
sudo ls -R /var/log/remote/

# The 'logger' program tag becomes the file name
sudo grep -r "shoreup-test" /var/log/remote/

You should see the line under something like /var/log/remote/<client-ip>/shoreup-test.log. If it's there, forwarding works.

If nothing arrives, work down the path:

# On the client: is the connection to the collector actually open?
sudo ss -tnp | grep ':514'

# On the collector: is anything arriving on the port? (needs tcpdump installed)
sudo tcpdump -n -i any tcp port 514

Also check both journals for rsyslog complaints:

journalctl -u rsyslog --since "10 minutes ago"

Common causes are a firewall blocking 514, a wrong target, or a config that failed validation so the service never applied it.

Set up rotation on the collector

Central logs grow fast. Add a logrotate rule so /var/log/remote doesn't fill the disk. Create /etc/logrotate.d/remote matching /var/log/remote/*/*.log, and after rotating, signal rsyslog to reopen its files with systemctl kill -s HUP rsyslog. Check the exact postrotate stanza against the logrotate manual for your distro before deploying — get the directive names right rather than guessing.

Undo

To stop forwarding on a client:

sudo rm /etc/rsyslog.d/60-forward.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog

The host reverts to local-only logging immediately.

To stop the collector receiving:

sudo rm /etc/rsyslog.d/10-remote-server.conf
sudo rsyslogd -N1 && sudo systemctl restart rsyslog
# then close the firewall port, e.g.  sudo ufw delete allow 514/tcp

Already-collected files under /var/log/remote/ stay put; remove that directory yourself only when you're sure you no longer need the logs — that deletion is irreversible.

For the authoritative parameter list — omfwd, imtcp, queue options, and templates — see the official rsyslog documentation (rsyslog.com/doc), and Debian's own rsyslog guidance for the file layout.

Written by
Ketan Aagja

Runs enterprise networks and security for a living, and writes Shore Up to turn two decades of hands-on Linux, Windows and mail-server work into guides you can actually use.

More about the author →

Was this article helpful?

Tap a star — no sign-in needed.

Be the first to rate this article.