The OpenSSL Memory Leak With No CVE to Patch Against
Okta's Red Team gave this one a name, HollowByte, and named things get attention. What deserves the attention here isn't the trick — it's the paperwork, or rather the lack of it.
What actually happens
A TLS handshake message starts with a four-byte header, three bytes of which announce how long the body is going to be. Older OpenSSL builds trusted that number on sight: the header lands, OpenSSL grows the receive buffer to the declared size, and only then waits for the body — before any of the handshake's own validation runs. For a ClientHello the ceiling is 131 KB. Send eleven bytes that promise a large body, then send nothing, and a worker thread sits blocked on a body that never comes. No auth, no session, no key exchange required. This is entirely pre-authentication.
On its own that's just a slow-loris variant, and we've defended against connection exhaustion for over a decade. The twist, as The Hacker News lays out, is glibc. When the attacker drops the connection OpenSSL frees the buffer, but glibc hangs onto small and medium chunks for reuse instead of handing them back to the kernel. By varying the claimed size on every connection, the attacker keeps the allocator from ever reusing what it freed. The heap fragments, RSS climbs, and it stays climbed after the attacker is long gone. Okta's NGINX tests OOM-killed a 1 GB box with 547 MB frozen in fragments, and pinned 25% of a 16 GB box without ever hitting the connection limit — which is exactly why, in their words, "standard connection-limiting defenses won't stop it."
Why this matters for a mail server specifically
Every TLS listener you run is in scope: Postfix smtpd on 25/465/587, Dovecot on 993/995, the Zimbra proxy. All of them present a TLS handshake to the internet before anyone authenticates, so an attacker needs no credentials and no valid mailbox.
Here's the honest nuance, though, and the source can't tell you this because it tested NGINX. Postfix runs a process-per-connection model — smtpd processes are spawned, recycled after max_use, and die. Dovecot's login processes behave similarly. When a process exits, its fragmented heap goes with it. That's genuinely different from NGINX's long-lived worker pool, where the frozen memory has nowhere to go until you restart. I'd expect the classic mail daemons to shed the damage faster than a persistent-worker web server does — but I haven't measured it, Okta didn't test it, and a determined flood can still starve the box while it's happening. Treat that as reasoning about your exposure, not a guarantee.
The real problem: nothing will tell you you're vulnerable
OpenSSL fixed this in June and decided it wasn't a vulnerability. They triaged it as "bug or hardening" only — a category that sits below the lowest of their four severity tiers. So there is no CVE, no advisory, and no changelog line pointing at it. That's the part that should bother you. Your scanner matches on CVE identifiers. Red Hat and the other downstreams backport fixes and leave the version string alone, and what normally resolves that ambiguity is the advisory and the OVAL feed — both keyed to CVE names that don't exist here.
So the check is manual. The fixed releases are 4.0.1, 3.6.3, 3.5.7, 3.4.6 and 3.0.21, all dated June 9. If you build OpenSSL yourself, move to the listed release and restart everything that loaded the old library. If you're on a distro package, you have to read the package changelog or ask the maintainer directly whether they rebased on the June 9 release or pulled the patch — that's PR 30792 for master and 4.0, 30793 for 3.6/3.5/3.4, and 30794 for 3.0.
Two caveats worth holding onto. Zimbra ships its own OpenSSL build, so distro patching won't touch it — you're waiting on Zimbra. And the fix covers TLS only; the DTLS handshake path still sizes its buffer from the peer's declared length, unfixed, in 4.0.1. That doesn't matter for SMTP or IMAP, but keep it in mind for anything DTLS-based on the same host.
The same June 9 release also closed 18 real CVEs, including a High-severity use-after-free in PKCS7_verify(). So there's a proper reason to take that update regardless of what you think of HollowByte's triage.
Sources
Runs enterprise networks and security for a living, and writes Shore Up to turn two decades of hands-on Linux, Windows and mail-server work into guides you can actually use.
More about the author →Was this article helpful?
Tap a star — no sign-in needed.
Be the first to rate this article.