HollowByte Turns TLS Buffer Growth Into Patch Work

Okta Red Team disclosed HollowByte on July 16, 2026, after reporting the issue to OpenSSL. In Okta's tests, a remote unauthenticated peer could send an 11-byte TLS input whose handshake header claimed a much larger body. Older OpenSSL code grew its receive buffer to the declared size before the body arrived. Repeated connections with varied sizes caused persistent heap fragmentation on the tested glibc-based NGINX servers and could push a small host into an out-of-memory condition.

OpenSSL handled the report as a bug or hardening fix, not a CVE security advisory. The incremental buffer-growth change shipped in OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 on June 9. That classification changes how teams must find the fix: a CVE-only scanner may have nothing to match. Operators should map exposed TLS services to their actual OpenSSL package, verify that the vendor build contains the relevant backport, deploy through the supported package path, and restart or redeploy every process that still has the older library loaded.

Okta says its Red Team found a denial-of-service path in OpenSSL's TLS handshake processing and demonstrated the effect against NGINX linked to vulnerable OpenSSL builds. The OpenSSL pull request confirms the core behavior: the old code grew init_buf immediately to the expected message size, and the change makes that growth incremental as data is received. The pull request also confirms that the report came from Okta Red Team.

The parties use different security labels. Okta calls HollowByte a denial-of-service vulnerability and recommends immediate package upgrades. The OpenSSL security team chose to handle it as a bug or hardening fix. OpenSSL therefore did not assign HollowByte a CVE or list it as a separate item in the June 9 security advisory. Both facts belong in an operational assessment; calling the issue a critical zero-day would go beyond the public record.

There is also no public evidence in the reviewed sources of active exploitation or data theft through this path. The demonstrated effect is service availability. It may still matter for internet-facing gateways, chat backends, API edges, VPN management surfaces, and small containers because a low-bandwidth remote input can impose much larger memory and worker costs on the receiving service.

A TLS handshake message begins with a four-byte handshake header. Three bytes declare the length of the body that should follow. Okta's example sends enough record and handshake framing to make an 11-byte input while declaring a much larger ClientHello body. In the older path, OpenSSL used that untrusted length to grow the initial receive buffer before the peer supplied the promised body.

Okta reports that the allocation could reach about 131 KB for the tested ClientHello path. The worker then waits for bytes that the peer never intends to send, subject to the application's socket and handshake timeouts. A single allocation is not remarkable. The asymmetry appears when an attacker opens many connections and varies the claimed sizes, creating repeated allocations and frees while sending very little actual data.

The merged OpenSSL change alters that accounting. The library grows the buffer in steps as new chunks arrive, with the code review noting at most eight reallocations. A peer that announces a large message but sends nothing no longer gets the full declared allocation at the start. This is a narrow and understandable mitigation, which makes it suitable for verification by downstream maintainers even without a CVE.

OpenSSL does free the buffer when the connection ends. Okta's central finding is what happened next on the glibc systems it tested. The allocator kept small and medium freed chunks available to the process rather than immediately returning them to the operating system. By changing claimed lengths across waves, the researchers reduced useful reuse, fragmented the heap, and kept the process's resident memory high after the connections closed.

In Okta's lab, an unpatched NGINX instance on a 1 GB system was killed after 547 MB became tied up in fragmented memory. On a 16 GB system, the test held about 25 percent of system memory while remaining under the configured connection ceiling. Those are vendor research results from specific builds and conditions, not universal capacity predictions. Different allocators, process models, timeouts, proxy layers, and operating systems can change both amplification and recovery.

Teams should therefore avoid two opposite mistakes. Dismissing the issue because every connection is bounded ignores the observed allocator effect. Treating every OpenSSL service as equally vulnerable ignores deployment conditions. A controlled test with production-like binaries, limits, and traffic handling can establish whether resident memory returns after malformed, incomplete handshakes without running an untrusted public exploit.

OpenSSL merged the main and 4.0 change in pull request 30792, the 3.6, 3.5, and 3.4 backports in pull request 30793, and the 3.0 backport in pull request 30794. The project's release timeline shows OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 released on June 9, 2026. Those are the upstream versions Okta identifies as containing the incremental-growth fix.

Do not turn that list into a naive string comparison on Linux distributions or appliances. Vendors often keep a stable base version and backport selected commits. An older-looking version can contain the fix, while a custom application image can carry its own vulnerable copy even after the operating system package is current. Ask the package or appliance vendor whether the build includes the commits from the relevant OpenSSL pull request, then record the package release and evidence in the exception or remediation ticket.

The public pull request says the change tackles TLS only. It explicitly leaves DTLS untouched because the corresponding work would be more invasive. That statement does not by itself prove an exploitable DTLS issue, but it does set the scope of the published remediation. Teams that expose DTLS should ask their supplier for a separate assessment rather than assuming the TLS change covers every protocol path.

Start at externally reachable TLS listeners, but trace ownership inward. Reverse proxies, API gateways, mail relays, collaboration servers, database endpoints, VPN portals, monitoring agents, language runtimes, containers, sidecars, and vendor appliances may bundle or dynamically load different OpenSSL copies on the same host. A shell-level openssl version command reports one executable, not necessarily the library used by every running process.

Use package-manager data, container SBOMs, image manifests, binary dependency inspection, vendor advisories, and runtime maps to connect each listener to a package or embedded library. Record whether TLS termination occurs at a load balancer before traffic reaches the application. If the front end terminates the hostile handshake, patching only an internal service will not address the exposed path. If a managed service owns termination, obtain provider confirmation rather than attempting unsupported changes.

After deploying the fixed package, restart or redeploy affected processes and verify the loaded library. Long-running workers can keep the old code mapped after files on disk change. Container hosts may also retain old images in a deployment, rollback set, job definition, or autoscaling template. A version check is complete only when the active process and the next replacement instance both use the corrected build.

If a supported fixed package is not yet available, use layered availability controls while escalating to the supplier. Set finite handshake and idle timeouts, cap concurrent incomplete handshakes per source and globally, enforce process or container memory limits with restart supervision, and put vulnerable origin services behind a patched TLS terminator where architecture and trust requirements allow. Rate limits should be tested because Okta's varied-size method was designed to create allocator pressure without simply exceeding a connection ceiling.

Monitor incomplete TLS handshakes, connection churn, allocator or resident-memory growth, worker saturation, out-of-memory events, and restart frequency together. A memory alert without connection context can look like an ordinary leak. A connection alert without process memory can miss the lasting effect Okta observed. Preserve samples of the malformed handshake metadata and load-balancer logs if the activity appears hostile, while avoiding full packet retention where it conflicts with privacy or volume constraints.

The missing CVE is itself a workflow lesson. Security programs that ingest only CVE feeds will miss some fixes that maintainers classify as hardening. Release-note review, upstream commit monitoring for foundational libraries, supplier attestations, and dependency ownership are needed alongside scanners. HollowByte does not prove that every unnumbered bug is urgent. It shows why externally reachable input paths deserve review even when the fix arrives without the metadata automation expects.