Shore Up
A magnifying glass held over a long paper scroll of stacked lines, with a few lines glowing and being tallied into two short ranked lists beside it.
Linux

Parse Nginx Access Logs for Top IPs and 404s with awk

Ketan Aagja10 min read
No ratings yet

Before you run this

Everything in this guide is read-only. The awk, sort, and uniq commands here only read the Nginx access log and print summaries to your terminal — they do not modify, rotate, or delete the log, and they change nothing on the server. There is nothing to undo.

You will usually need read access to the log file. On Debian/Ubuntu, /var/log/nginx/access.log is typically owned by root and readable by the adm group, so run these commands with sudo, or add your user to the adm group if that fits your policy. If you can already cat the log, you don't need root.

Even though nothing here is destructive, test on a copy first if you're new to the log format. Copy a few thousand lines to your home directory (cp /var/log/nginx/access.log ~/access-test.log) and run the commands against that until the output looks right. The one real risk is drawing wrong conclusions from a misread field — for example, if your Nginx uses a custom log_format, the field positions below will be wrong and you'll count the wrong column. So confirm your format before you act on the numbers (see "Check your log format" below).

Assumptions for this guide:

  • Debian 12 / Ubuntu 22.04 with Nginx installed from the distro package.
  • The default combined log format, which Nginx uses out of the box.
  • Log at /var/log/nginx/access.log, with rotated files as access.log.1, access.log.2.gz, and so on.
  • A standard bash shell with GNU awk, sort, and uniq (all present by default).

If you're on RHEL/AlmaLinux, the path and format are the same; the log group may be nginx or root rather than adm.

Check your log format

Every command below depends on knowing which whitespace-separated field is which. The default combined format is:

log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

Confirm your server actually uses it:

grep -R "log_format" /etc/nginx/            # shows any custom formats defined
grep -R "access_log" /etc/nginx/            # shows which format each site uses

If access_log lines end in just a path (no format name), Nginx is using combined. A sample line looks like this:

192.0.2.10 - - [10/Oct/2024:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 612 "-" "Mozilla/5.0"

Split on spaces, that gives us the fields we care about:

  • $1 — client IP (remote_addr)
  • $6 — request method, with a leading quote ("GET)
  • $7 — request path (/index.html)
  • $9 — HTTP status code (200)

Those positions hold for combined. If you use a custom format, count your own fields from a real line before trusting anything below — the request field being quoted is exactly why the path lands in $7.

Top IPs by request count

The classic pipeline: pull the first field, sort it, collapse duplicates with a count, then sort by that count descending.

sudo awk '{print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

Output is <count> <ip>, most-active first:

  48213 192.0.2.10
  11902 198.51.100.7
   9048 203.0.113.44

You can do the same entirely in awk with an associative array, which is faster on large files because it avoids sorting the whole stream:

# Tally each IP in one pass, then sort only the summary
sudo awk '{count[$1]++} END {for (ip in count) print count[ip], ip}' \
  /var/log/nginx/access.log \
  | sort -rn | head -20

Both give the same answer. I reach for the first form for a quick look and the second when the file is hundreds of megabytes.

Top 404s (which URLs are missing)

Filter on the status field being 404, then rank the requested paths:

# $9 is the status code; $7 is the requested path
sudo awk '$9 == 404 {print $7}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

This is the one I run most. It tells you what people (and bots) are asking for that isn't there — dead links, moved assets, or someone probing for /wp-login.php on a server with no WordPress.

To see who is generating the 404s rather than what they're hitting, print the IP instead:

sudo awk '$9 == 404 {print $1}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

And to see the IP and the path together — useful for spotting a single host walking your URL space:

sudo awk '$9 == 404 {print $1, $7}' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn | head -20

A quick status-code overview

Before drilling into 404s, it's often worth a one-line census of every status code so you know the shape of the traffic:

sudo awk '{count[$9]++} END {for (code in count) print count[code], code}' \
  /var/log/nginx/access.log \
  | sort -rn

If you see a wall of 499 (client closed connection) or 502/504, your problem is upstream, not missing files.

Including rotated and gzipped logs

The commands above only read the current access.log. To cover a full retention window, include the rotated files. Plain rotated files can be added to the awk argument list; gzipped ones need zcat. The simplest reliable approach is to run two passes, or use zcat for everything since it handles both:

# Uncompressed rotated logs alongside the current one
sudo awk '$9 == 404 {print $7}' /var/log/nginx/access.log /var/log/nginx/access.log.1 \
  | sort | uniq -c | sort -rn | head -20

# Gzipped archives — decompress on the fly
sudo zcat /var/log/nginx/access.log.*.gz \
  | awk '$9 == 404 {print $7}' \
  | sort | uniq -c | sort -rn | head -20

Note that piping from zcat means awk reads from standard input, so you drop the filename argument. On some systems zcat refuses files that don't end in .gz; zcat -f (GNU) reads plain files too, but keep the passes separate if you're unsure.

A small reusable script

If you run these often, drop them in a script that takes the log path as an argument. This one is read-only and prints all three summaries:

#!/usr/bin/env bash
# nginx-report.sh — quick top-IPs / status / 404 summary for a combined-format log
# Usage: sudo ./nginx-report.sh /var/log/nginx/access.log
set -euo pipefail

LOG="${1:-/var/log/nginx/access.log}"   # default to the standard path

echo "== Top 20 IPs =="
awk '{count[$1]++} END {for (i in count) print count[i], i}' "$LOG" \
  | sort -rn | head -20

echo; echo "== Status codes =="
awk '{c[$9]++} END {for (s in c) print c[s], s}' "$LOG" | sort -rn

echo; echo "== Top 20 404 URLs =="
awk '$9 == 404 {print $7}' "$LOG" | sort | uniq -c | sort -rn | head -20

Make it executable with chmod +x nginx-report.sh. Replace the default path if your log lives elsewhere.

Verify the output is trustworthy

Because a misread format gives confident but wrong numbers, sanity-check against a tool that parses the log independently:

# Total lines in the log
wc -l /var/log/nginx/access.log

# Total 404s two ways — the counts should match
grep -c ' 404 ' /var/log/nginx/access.log
awk '$9 == 404' /var/log/nginx/access.log | wc -l

The grep -c ' 404 ' count is a rough cross-check (a 404-byte response size could theoretically match, which is why the space-padded pattern helps); the awk count is the authoritative one for the status field. If the top-IP counts sum to roughly your total line count, your field positions are right. If the two 404 numbers are wildly different, your log isn't in combined format — go back to "Check your log format" and recount the fields.

For deeper or ongoing analysis, GoAccess reads the same logs and renders live dashboards, but for a fast, dependency-free answer on a box you're already logged into, awk is hard to beat.

Written by
Ketan Aagja

Runs enterprise networks and security for a living, and writes Shore Up to turn two decades of hands-on Linux, Windows and mail-server work into guides you can actually use.

More about the author →

Was this article helpful?

Tap a star — no sign-in needed.

Be the first to rate this article.