Photo by FlyD on Unsplash

SYN Cookies: Stateless TCP Defense Against SYN Floods


The SYN Flood Problem

When a TCP connection starts, the client sends a SYN packet, the server responds with SYN-ACK, and the client completes the handshake with ACK. The server allocates memory for connection state after receiving the SYN and before the handshake completes. This creates an attack surface: an attacker can send thousands of SYN packets with spoofed source addresses, forcing the server to maintain half-open connections that will never complete. The server’s connection queue fills up, and legitimate connections get dropped.

The traditional defense involves tuning queue sizes and timeouts, but these are band-aids. A sufficiently large attack will still exhaust server resources. The real problem is that the server commits memory before the client proves it can receive responses.

Encoding State in the Sequence Number

SYN cookies solve this by eliminating server-side state during the handshake. Instead of storing connection information in memory, the server encodes it cryptographically into the initial sequence number sent in the SYN-ACK packet.

When the server receives a SYN, it computes a sequence number by hashing the connection tuple (source IP, source port, destination IP, destination port) along with a secret key and timestamp. This hash becomes the ISN (initial sequence number) in the SYN-ACK. The server then discards all connection state.

If the client is legitimate and completes the handshake, it sends back an ACK with the sequence number incremented by one. The server can then recompute the hash from the connection tuple and verify that the ACK matches. If it does, the connection is valid and the server allocates resources. If not, the packet is dropped.

Encoding Connection Options

The basic SYN cookie technique works, but it has a limitation: TCP options negotiated during the handshake (like window scaling, timestamps, and selective acknowledgment) need to be remembered. The sequence number has limited bits available for encoding.

Most implementations reserve a few bits in the sequence number for the most critical option: maximum segment size (MSS). The client advertises its MSS in the initial SYN, and the server encodes a small integer representing the MSS value into the cookie. When the ACK returns, the server can extract this value and configure the connection appropriately.

Other options are either approximated, disabled, or retrieved by having the server send its own options in the SYN-ACK and hoping the client echoes them in the ACK. This is one of the tradeoffs: SYN cookies reduce functionality slightly to maintain availability.

Timestamps and Time Windows

The cryptographic hash includes a timestamp to prevent replay attacks. An attacker who captures a valid ACK packet shouldn’t be able to replay it hours later to establish connections. The timestamp is typically quantized to intervals of 30-60 seconds, which means a valid ACK must arrive within a short window of when the SYN-ACK was sent.

This creates a subtle side effect: connections that take longer than the time window to complete will be rejected. In practice, this is rarely a problem because legitimate clients respond within milliseconds, not minutes. The time window is chosen to be long enough for real-world network delays but short enough to limit replay attack utility.

Performance Characteristics

SYN cookies have minimal overhead when not under attack. The server still processes every SYN, but computing a hash is cheaper than allocating connection state. The real benefit appears during an attack: memory usage remains constant rather than growing with the attack size.

The stateless nature does introduce CPU overhead. Every incoming packet must be validated by recomputing the hash, whereas normal TCP can simply look up existing state in a hash table. However, modern cryptographic hash functions are fast enough that this rarely becomes a bottleneck.

When to Enable

Most operating systems support SYN cookies but don’t enable them by default. Linux, for example, activates them automatically when the SYN queue reaches a threshold. This hybrid approach preserves full TCP semantics under normal load while providing DDoS protection when needed.

The alternative strategy is always-on SYN cookies, which trades slightly reduced functionality for consistent behavior and maximum resilience. Cloud load balancers and DDoS mitigation services often take this approach since they prioritize availability over perfect TCP compliance.

The Broader Pattern

SYN cookies are part of a broader pattern in systems design: moving validation to the edges and making protocols stateless when possible. By not trusting the client until it proves it can receive responses, and by encoding necessary state in the protocol itself rather than server memory, the technique converts a resource exhaustion attack into a computational problem the server can handle.

This same principle appears in other contexts: stateless authentication tokens, client-side session storage, and challenge-response protocols all shift state management to avoid server-side resource commitment before validation.