VPN Protocols Compared: IPsec, WireGuard, TLS-Based and MACsec

Four ways to encrypt traffic between systems, operating at different layers with different trade-offs. Choosing between them is mostly about where the encryption boundary sits and what has to pass through in between.

IPsec

Network layer, so it protects everything above it without applications knowing. Two modes — transport for host to host, tunnel for gateway to gateway — and ESP as the protocol providing confidentiality and integrity, as covered under IPsec transport and tunnel mode.

Its strengths are ubiquity and interoperability: every serious network device supports it, and it is the default for site-to-site connectivity between different vendors' equipment.

Its weaknesses are complexity and traversal. The configuration surface is large — two negotiation phases, many algorithm choices, traffic selectors that must mirror — which is why the failures under VPN troubleshooting are so common. And ESP is IP protocol 50 rather than a port, so it needs NAT traversal to cross translation, and restrictive networks frequently block it.

WireGuard

Also network layer, and deliberately minimal. A fixed set of modern cryptographic primitives with no negotiation — no cipher suites to choose, no downgrade to negotiate — and a codebase small enough to be audited in an afternoon, which is a genuine security property rather than a marketing claim.

Configuration is a public key and an allowed address range per peer, which removes most of the ways IPsec is misconfigured. It runs over UDP on a single configurable port, traverses NAT naturally, and reconnects seamlessly when a client changes network because it is connectionless by design.

The trade-offs are real. No algorithm agility means a future weakness requires a protocol version change rather than a configuration change. It assigns static addresses per peer, which suits some designs and complicates large dynamic populations. Enterprise features — per-user authentication, policy, logging — come from products layered on top rather than from the protocol.

TLS-based VPNs

Transport layer, tunnelling traffic inside a TLS session. The decisive advantage is traversal: it looks like ordinary HTTPS on port 443, so it works from hotel networks, guest Wi-Fi and restrictive corporate networks that block everything else.

That is why TLS-based VPNs dominate remote access while IPsec dominates site-to-site. It also means the concentrator is an internet-facing web service, which has been a repeated source of exploited vulnerabilities — the exposure noted under remote access VPN design.

Performance over TCP suffers from the layered-retransmission problem, where the inner and outer protocols both retransmit and interact badly, so implementations offering a UDP mode use it by preference.

MACsec

Layer 2, encrypting Ethernet frames on a single link between two directly connected devices. Different job from the others: it protects a cable rather than a path.

The use cases are physical link protection where the cable leaves your control — between buildings, through a shared riser, across a provider's Ethernet service — and switch-to-switch encryption within a data centre where regulation requires encryption in transit everywhere.

Because it is hop-by-hop, each switch decrypts and re-encrypts, which means the traffic is in plaintext inside every device along the path. It is not end-to-end, and confusing it with an end-to-end control is the mistake. Its advantage is that it operates at line rate in hardware with negligible latency, which the higher-layer options cannot match.

Choosing

Site-to-site between different vendors: IPsec, because interoperability decides it.

Remote access for users on arbitrary networks: TLS-based, because traversal decides it — or increasingly a zero trust broker rather than a VPN at all.

Modern infrastructure you control on both ends, especially machine to machine: WireGuard, for simplicity and performance.

A physical link you do not trust: MACsec, alongside whatever else runs above it.

Note also what none of them provide. A VPN grants a network position and authenticates once at connection; it does not authorise what happens afterwards, which is the structural limitation that zero trust access addresses by authenticating per application instead.

And all four rely on key exchange for forward secrecy, so the properties discussed under forward secrecy apply to each — including the harvest-now-decrypt-later consideration for long-lived sensitive traffic.

Exam angles

Place IPsec at the network layer with transport and tunnel modes, widely interoperable and complex to configure with NAT traversal required. Know WireGuard as a minimal network-layer protocol with fixed cryptography and no negotiation. Know TLS-based VPNs as operating over port 443 for traversal, which makes them standard for remote access. Know MACsec as hop-by-hop layer 2 link encryption, not end-to-end, operating in hardware at line rate.