DTLS vs TLS: Why UDP Needs a Different Handshake

TLS assumes the transport beneath it is reliable and ordered. TCP guarantees both, so TLS can treat its handshake as a conversation where nothing is lost and nothing arrives out of sequence. Take that guarantee away and TLS breaks outright — not because the cryptography fails, but because the protocol has no way to recover from a dropped message.

DTLS is TLS adapted to run over UDP. Same cipher suites, same certificates, same security guarantees; different plumbing.

Why UDP Needs Anything Different

Three problems appear the moment you remove TCP.

Messages get lost. TLS would simply stall. DTLS adds its own retransmission timer for handshake messages, so a lost ServerHello is resent rather than hanging the connection.

Messages arrive out of order. DTLS numbers each handshake message with an explicit sequence number so the receiver can reassemble them correctly.

Records cannot depend on each other. In TLS, the stream cipher state carries across records. DTLS makes each record independently decryptable and adds an explicit sequence number to every record, so losing one does not poison the rest.

The Cookie Exchange

This is the piece most worth remembering, because it exists for a security reason rather than a reliability one.

UDP has no handshake, so the source address in a datagram is trivially forged. Without protection, an attacker could send a small ClientHello with a spoofed source and make the server perform expensive cryptography and send a much larger response to a victim — a straightforward amplification attack of the kind described in distributed reflected denial of service.

DTLS inserts a stateless cookie exchange. The server responds to the first ClientHello with a HelloVerifyRequest containing a cookie; the client must echo it in a second ClientHello. The server allocates no state and does no real work until that round trip completes, which proves the client can actually receive at the claimed address.

Fragmentation

TLS handshake messages, especially certificate chains, routinely exceed the path MTU. TCP fragments transparently. UDP does not, and IP-level fragmentation is unreliable across the internet.

DTLS therefore fragments handshake messages at its own layer, giving each fragment an offset and length so the receiver can reassemble them. This is also why large certificate chains cause more trouble on DTLS than on TLS, and why compact certificates are worth having.

What Stays the Same

Everything that matters for security. The same cipher suites, the same certificate validation, the same key exchange. DTLS 1.2 corresponds to TLS 1.2 and DTLS 1.3 to TLS 1.3, including the reduced round trips and the mandatory ephemeral key exchange that provides perfect forward secrecy.

Anti-replay works differently, though: since UDP genuinely does deliver duplicates, DTLS uses a sliding receive window over record sequence numbers, in much the same way IPsec does.

Where DTLS Is Actually Used

• WebRTC — browser voice and video. DTLS performs the handshake and derives the keys that SRTP then uses for the media itself.
• SIP over UDP — secured signalling where TCP is not in use; see Session Initiation Protocol.
• OpenVPN in UDP mode and Cisco AnyConnect's DTLS channel, which exists specifically to avoid running TCP inside TCP.
• CoAP for constrained IoT devices, where a full TCP stack is unwelcome.
• QUIC is the exception. It also runs over UDP but does not use DTLS; it embeds TLS 1.3 directly into its own transport. Expect this as a distractor.

Why Not Just Use TCP

For real-time media, retransmission is worse than loss. A voice packet that arrives 400 milliseconds late is useless, and TCP's insistence on delivering it in order stalls everything behind it — head-of-line blocking. Tunnelling TCP inside TCP compounds the problem, since two retransmission timers fight each other and throughput collapses under loss.

UDP lets the application decide what to do about loss. DTLS lets it make that decision without giving up encryption.

Exam Focus

Know DTLS provides the same security as TLS over an unreliable transport. Know the cookie exchange exists to prevent denial-of-service amplification from spoofed source addresses. Know DTLS adds retransmission, sequence numbers and its own fragmentation. Associate it with WebRTC, VPN data channels and IoT. And remember QUIC uses TLS 1.3 directly rather than DTLS.