DTLS: UDP Gets Its Own TLS Handshake


TLS was designed around TCP’s reliability guarantees: ordered delivery, retransmission, and a stable connection. But UDP applications like WebRTC, VoIP, online gaming, and IoT protocols need encryption too, and they can’t afford TCP’s latency overhead or head-of-line blocking. DTLS (Datagram Transport Layer Security) solves this by adapting TLS cryptography to work over unreliable, unordered transport.

Why TLS doesn’t work over UDP

TLS assumes the underlying transport delivers bytes in order and reliably. The handshake is a strict sequence of messages, each depending on the previous one. If a packet is lost, TCP retransmits it transparently. TLS never has to handle missing or reordered messages—it just writes to a stream.

UDP offers no such guarantees. Packets can arrive out of order, be duplicated, or disappear entirely. If you tried to run standard TLS over UDP, a single lost handshake packet would stall the entire connection indefinitely. The application would have no way to know whether a message was lost or just delayed.

WebRTC, VPN protocols, and real-time media can’t tolerate TCP’s retransmission delays. They need encryption, but they also need to keep moving forward even when packets are lost. That’s the gap DTLS fills.

Handshake changes for unreliability

DTLS reuses TLS 1.2 and 1.3 cryptography—cipher suites, certificate validation, key derivation—but redesigns the handshake to tolerate packet loss. Every DTLS handshake message includes an explicit sequence number and fragment offset, so the receiver can detect gaps and reorder messages.

When a client sends a ClientHello and doesn’t receive a ServerHello within a timeout, it retransmits. The same applies to every handshake flight. DTLS groups messages into flights and retransmits the entire flight if any part is lost. This timeout-and-retransmit logic is built into the protocol itself, not delegated to the transport layer.

DTLS also adds a stateless cookie exchange to defend against DoS amplification attacks. An attacker can spoof UDP source addresses and trigger expensive cryptographic operations on the server. DTLS requires the client to echo a cookie before the server commits resources to the handshake, proving the client can receive packets at the claimed address.

Record layer differences

Once the handshake completes, DTLS encrypts application data just like TLS, but each record is independent. There’s no stream abstraction. Each UDP datagram carries a DTLS record with its own sequence number. The receiver uses the sequence number to detect replayed or reordered packets, but it doesn’t wait for missing records—it processes whatever arrives.

This means DTLS can’t hide packet boundaries. Each encrypted datagram maps to one application message. If your application sends small messages, you’ll get small packets. There’s no stream buffering or coalescing across records.

DTLS also has to handle the Maximum Transmission Unit (MTU). Large handshake messages like certificates can exceed the path MTU, so DTLS fragments them and includes reassembly metadata. The receiver buffers fragments and reassembles them before processing. This adds complexity but keeps the handshake from fragmenting at the IP layer, which would be catastrophic for reliability.

When DTLS makes sense

DTLS is the right choice when you need TLS-grade security over UDP, but it’s not free. You take on complexity: timeout tuning, replay detection, MTU handling, and handshake retransmission logic. Libraries handle most of this, but debugging DTLS issues in production requires understanding how the protocol compensates for UDP’s unreliability.

WebRTC uses DTLS to secure media streams and data channels. OpenVPN and WireGuard alternatives use it for VPN tunnels. CoAP, the IoT protocol, layers on DTLS for constrained devices that can’t afford TCP overhead.

If your application can tolerate TCP’s latency characteristics, TLS over TCP is simpler. But when you need the control and low latency of UDP with modern encryption, DTLS is the protocol that makes it possible. It’s proof that security protocols can adapt to transport constraints without compromising cryptographic guarantees.