Why VPN Tunnels Fail: Phase 1, Phase 2, NAT-T and MTU

IPsec VPN troubleshooting is mostly a matter of knowing which of two negotiations failed, because the causes are entirely different. Establish that first and the rest narrows quickly.

The two phases

Phase 1 authenticates the peers and builds a secure channel for the negotiation itself. Both sides must agree on the encryption and hash algorithms, the Diffie-Hellman group, the authentication method, and the lifetime. It produces a management channel, and no user traffic flows over it.

Phase 2 negotiates the security associations that actually carry data: which traffic is protected, with which algorithms, and for how long.

The diagnostic value is that a phase 1 failure means the peers never agreed on how to talk at all — look at algorithms, the pre-shared key or certificate, and peer addresses. A phase 2 failure means authentication succeeded and they disagree about what to protect — look at the traffic selectors.

IKEv2 collapses much of this and handles reconnection, mobility and rekeying far better than IKEv1's main and aggressive modes. Use it where both ends support it; the phase concepts remain, and the failure modes below are the same. The wider context is under IPsec transport and tunnel mode.

Phase 1 failures

Mismatched proposals. One side offers AES-256 with SHA-256 and group 14; the other expects AES-128 with SHA-1 and group 5. Nothing overlaps, and negotiation fails with a message that rarely names the specific mismatch. Compare the full proposal on both sides line by line — this accounts for most phase 1 problems.

Authentication mismatch. A pre-shared key with a typo or a trailing space, or a certificate whose subject does not match what the peer expects, or an expired certificate. Where certificates are used, both peers must also trust the issuing authority and be able to check revocation, which is where the fail-open behaviour noted under PKI structure becomes relevant.

Peer identity mismatch. Behind NAT or with dynamic addressing, the identity a peer presents may not match the address it arrives from, and the configuration has to account for that explicitly.

Clock skew breaks certificate validity checks, which is one of the dependencies described under NTP slew and step.

Phase 2 failures

Almost always the traffic selectors, sometimes called proxy IDs. Both peers must agree on exactly which networks are protected, and the definitions must mirror each other — local and remote reversed on the far side.

A common mismatch is one side configured with a summarized range and the other with several specific subnets. They overlap logically and do not match as selectors, so phase 2 fails while phase 1 shows established. Some implementations accept a broader selector and many do not.

Algorithm and lifetime mismatches apply here too, and perfect forward secrecy must be enabled or disabled consistently — one side requiring a fresh key exchange while the other does not is a silent mismatch.

NAT traversal

The classic case of a tunnel that works between two public addresses and fails from behind a router.

ESP is IP protocol 50, not a TCP or UDP port, so a translating device has no port number to work with. NAT traversal detects NAT in the path during phase 1, moves the negotiation to UDP 4500, and encapsulates ESP inside UDP so translation works.

So the checks are: NAT-T enabled on both sides, UDP 500 and UDP 4500 permitted through every firewall in the path, and keepalives configured so the translation table entry does not expire during idle periods — which produces the maddening symptom of a tunnel that works and then stops after a few minutes of silence.

Note also that AH cannot cross NAT at all, because it protects the header fields translation modifies, which is one reason ESP is used almost universally — and the translation mechanics are those under NAT and PAT.

Established but not passing traffic

A distinct category, and the one people spend longest on.

Routing. The tunnel is up and nothing sends traffic into it. Check that a route points the protected networks at the tunnel interface, and that return traffic on the far side does the same.

Firewall policy. Encryption is not authorization — a rule must permit the traffic after decryption, and on many platforms the tunnel interface is a separate zone that needs its own policy.

MTU and fragmentation. Encapsulation adds overhead, so a full-size packet becomes too large. Small packets pass and large transfers hang, which is the recognizable signature — and it is worse when ICMP fragmentation-needed messages are blocked, since path MTU discovery then fails silently. Clamping the TCP maximum segment size on the tunnel interface is the standard fix.

Rekey failures produce a tunnel that works for exactly the lifetime interval and then drops. Mismatched lifetimes are the usual cause, and the shorter one wins if the implementations agree to negotiate it.

Exam angles

Know phase 1 as authenticating peers and establishing the management channel, and phase 2 as negotiating the security associations protecting data. Diagnose by which phase failed: phase 1 for algorithm, key, certificate and identity mismatches; phase 2 for traffic selector and PFS mismatches. Know UDP 500 for IKE and UDP 4500 for NAT traversal, ESP as IP protocol 50, and that AH cannot cross NAT. Recognize MTU as the cause when a tunnel is up but large transfers fail.