IPsec Transport vs Tunnel Mode: AH, ESP, and Where Each Belongs

IPsec secures traffic at the network layer, which means it protects everything above it without any application needing to know. Two choices define any deployment: which mode — transport or tunnel — and which protocol — AH or ESP. The exam tests both, and the combinations behave quite differently.

Transport mode

Transport mode protects the payload of the original IP packet and leaves the original header in place. Source and destination addresses stay visible, and the packet routes exactly as it would have.

That makes it appropriate for end-to-end protection between two hosts that are already routable to each other: a management workstation to a server, a server to a database, or host-to-host policies inside a data center. There is no encapsulation overhead beyond the IPsec header itself, so it is the more efficient of the two.

Its limitation follows from the same property. Because the original addresses are exposed, transport mode provides no topology hiding, and because there is no new outer header, it cannot carry traffic on behalf of networks behind a gateway.

Tunnel mode

Tunnel mode encapsulates the entire original packet — header included — inside a new IP packet addressed to the peer gateway. The internal addressing is hidden from anyone watching the path, and the gateways handle protection on behalf of the hosts behind them.

This is what site-to-site VPNs use, and what remote access VPNs use, because the client's traffic must be carried into a network it is not natively part of. The cost is the extra header on every packet, which reduces the effective MTU and makes fragmentation a recurring operational annoyance.

The rule of thumb: if a gateway is protecting traffic that is not its own, it is tunnel mode. If two endpoints are protecting their own conversation, transport mode is available.

AH and ESP

Authentication Header provides integrity and origin authentication over the packet, including most of the IP header, plus replay protection. It provides no confidentiality — the data is signed, not encrypted.

Because AH covers the outer header fields, it breaks the moment anything rewrites an address. A device performing NAT or PAT changes exactly the fields AH is protecting, so the integrity check fails at the far end. AH cannot traverse NAT, and that single fact is why it is rarely deployed today.

Encapsulating Security Payload provides confidentiality plus integrity and authentication of the payload, without covering the outer header. That omission is what lets it survive address translation, and it is why essentially every real deployment uses ESP.

Modern practice is ESP with an authenticated encryption algorithm, which supplies both properties in one operation. Combining AH and ESP is possible and almost never worth the complexity.

IKE and the ports to know

Internet Key Exchange negotiates the security associations, authenticating the peers and establishing the keys. IKEv2 is the current version and handles reconnection, mobility, and rekeying far better than IKEv1's main and aggressive modes.

The port numbers appear on exams regularly. IKE uses UDP 500. NAT traversal moves the exchange to UDP 4500 and wraps ESP inside UDP so translation devices can handle it, since raw ESP is IP protocol 50 and AH is protocol 51 — protocol numbers, not ports, which is precisely why NAT struggles with them. When an IPsec tunnel works between two public addresses and fails from behind a router, NAT traversal is the first thing to check.

Authentication between peers is normally a pre-shared key or a certificate. Pre-shared keys are simple and scale badly; certificates scale and require a PKI.

Where it fits now

IPsec remains standard for site-to-site connectivity between offices and to cloud providers. For remote access it competes with TLS-based VPNs, which pass through restrictive networks more easily because they look like ordinary web traffic, and increasingly with zero trust network access, which authenticates per application rather than placing a device on the network.

IPsec was designed as an integral part of IPv6 rather than an add-on, so a dual stack environment needs its IPsec policy defined for both families — a policy written only for IPv4 leaves the IPv6 path unprotected, which is the same blind spot that shows up throughout dual-stack security.

Exam angles

Transport mode protects the payload and keeps the original header, used host to host. Tunnel mode encapsulates the whole packet in a new one, used gateway to gateway and for remote access. AH gives integrity and authentication with no encryption and cannot cross NAT; ESP gives confidentiality plus integrity and can. Know UDP 500 for IKE, UDP 4500 for NAT traversal, and protocols 50 and 51 for ESP and AH. When a scenario needs internal addressing hidden, the answer is tunnel mode with ESP.