Detection bench

The question

Most detection content shows a query that catches the technique. Far less of it shows what the same query does on a normal Tuesday. A rule with a hundred percent detection rate and a hundred false positives a day gets muted within a week, which makes it worse than no rule at all — someone believes it is covering them.

So every rule here gets two numbers: does it catch the thing, and how often does it catch something else.

Setup

The bench has three parts. A scenario generator runs a known technique in the lab tenant. A noise generator runs plausible benign activity that looks structurally similar — the administrator who genuinely does run unusual commands, the service account that genuinely does authenticate from odd places. Then the KQL runs across both windows and the harness records what it matched.

The benign traffic is the hard part and the part most worth building. It is easy to write a detection that fires on the attack; it is the near-misses that decide whether a rule survives.

What gets recorded

• True positives against the simulated technique, by variant.
• False positives against the benign window, with the specific benign action that triggered each one.
• Query cost — a detection that scans too much is a detection that gets turned off.
• How the rule degrades when a field is missing, since real telemetry is patchy.

Working notes

• Most of the tuning work is not in the detection logic, it is in the exclusions. Which means the exclusions need to be documented as carefully as the rule.
• Detections built on process names age badly. Detections built on behaviour and relationships survive a rename.
• A rule that cannot be explained to the person who has to triage it will not get triaged correctly, no matter how elegant the query is.

Where it goes next

Extending the noise generator so it produces a realistic baseline over days rather than hours, which is the only honest way to measure a rule that keys on rarity.