XMAS, FIN and NULL Scans: TCP Flag Scanning and Why It Often Fails

XMAS, FIN and NULL scans are a family of TCP probes that exploit a gap in the specification rather than a vulnerability. They are worth knowing for the exam and worth understanding honestly: the technique is clever, frequently unreliable, and easy to detect.

The behaviour they exploit

The TCP specification says that a closed port receiving a segment without the SYN, ACK or RST flags set should reply with RST. It also says an open port in the listening state should ignore such a segment entirely.

That asymmetry is the whole technique. Send a deliberately odd packet and interpret the answer: a RST means the port is closed, and silence means the port is open — or filtered, which is the problem discussed below.

The three variants differ only in which flags they set. An XMAS scan sets FIN, PSH and URG, named because the packet is "lit up like a Christmas tree." A FIN scan sets only FIN. A NULL scan sets no flags at all. All three produce the same inference.

Why the results are ambiguous

Open and filtered are indistinguishable. A firewall that silently drops packets to a closed port produces exactly the same silence as an open port, so the scanner reports open|filtered and you do not actually know which.

Worse, the specification is not universally implemented. Windows systems reply with RST to these probes regardless of port state, so every port appears closed and the scan returns nothing useful. Several other stacks behave inconsistently too. A scan that reports all ports closed against a host that is clearly running services is usually a stack that does not follow the rule — not evidence about the ports.

So the practical position is that these scans are useful against certain older or Unix-derived stacks and unreliable elsewhere. A standard SYN scan gives clearer answers in almost every case, as covered under high-speed port scanning.

The stealth claim

These are often described as stealth scans, and the claim needs qualifying. They were designed to slip past simple stateless packet filters that only inspected SYN packets, and against a firewall of that generation they worked.

Against anything modern they do not. A stateful firewall drops a FIN packet with no matching connection as invalid, and an intrusion detection system flags illegal flag combinations immediately — a NULL packet with no flags cannot occur in legitimate traffic, so it is an unambiguous signature. The odd combinations that made these scans work are precisely what makes them obvious.

They avoid appearing in application logs, since no connection is established, which is the one part of the stealth claim that survives. Firewall and sensor logs see everything.

Where they remain useful

Two legitimate uses. Firewall rule testing: comparing how a filter responds to SYN, FIN and NULL probes reveals whether it is stateful and which flag combinations it passes, which is information a straightforward scan does not provide. Operating system fingerprinting: because implementations differ in how they respond, the pattern of responses helps identify the stack, which is one input among many.

Idle scanning, which uses a third host's IP identification counter to infer port state without sending anything from your own address, belongs to the same family of specification-behaviour tricks and is the one with a genuine anonymity property — at the cost of needing a suitable idle host.

Defending against them

A stateful firewall handles this by design: a packet with FIN set and no established connection has nothing to match and is dropped. Default-deny inbound removes most of the surface regardless of flags.

Detection is straightforward because the packets are illegal. Alert on NULL packets, on FIN without an established session, and on flag combinations such as SYN together with FIN that cannot occur legitimately. These are low-false-positive rules and worth having in place rather than tuning later.

Correlate across destinations rather than counting per host — the pattern is one source touching many ports or many addresses, which shows up in firewall logs collected into your SIEM. Unused address space is again the highest-confidence sensor, since nothing legitimate ever probes it.

Note that reconnaissance detection tells you someone is interested rather than that you are compromised. Treat a scan as an indicator to check exposure and patch status on whatever was probed, not as an incident in itself — and remember that public scan indexes mean an attacker can profile an internet-facing host without ever sending you a packet, as discussed under internet-wide scanning.

Exam angles

Know XMAS (FIN, PSH, URG), FIN (FIN only) and NULL (no flags), and the inference rule: RST means closed, no response means open or filtered. Recognize that Windows does not follow the specification, so these scans are unreliable against it. Understand that stateful firewalls block them and intrusion detection flags illegal flag combinations, so the stealth advantage applies only to legacy stateless filters and to application logs.