Mutation testing and AI
Why green tests are not enough, and how agents can close the loop
Two articles, born out of a talk, on what having reliable tests really means. It starts from the difference between coverage and mutation score, with a real bug that stayed in production for three weeks, and gets to closing the loop: agents that read the mutant report to improve themselves, with the first production numbers (Meta ACH and Just-in-Time testing).
Green tests and healthy code are not the same thing
We take an equation for granted: the tests pass, so the code works. Almost always it holds. When it does not, the bill arrives on code that is covered, tested and green, and no test had noticed.
Coverage answers one question only: does this code get executed? It does not answer the one that matters: if this code were wrong, would the tests notice? Mutation testing answers the second, by breaking the code on purpose and counting how many breakages the tests catch.
The series starts from a bug that stayed three weeks in production with 93% coverage, and gets to the question AI agents make urgent: if a machine writes the tests, who checks that they work? The answer is that you need an external, formal referee, and the mutation score can play that role.
What you will learn
- ✓ Tell coverage (the code gets executed) apart from mutation score (the tests actually verify)
- ✓ Recognize the typical holes in a suite: non-discriminating data, incomplete assertion, uncovered boundary
- ✓ Use the report of survived mutants as a feedback prompt for an agent
- ✓ See where mutation testing is entering production (Meta ACH, Just-in-Time testing)
Articles in the series
- 01How mutation testing showed me my suite was lying 7 min
93% coverage, all green, and a third of the tests verified nothing. What Stryker.NET showed, and how I went from 65% to 92%.
→ - 02Agents write the tests. Who checks that they work? 7 min
Mutation score correlates with real bugs at R² ≈ 0.70, line coverage stops at 0.25. How it becomes the referee for agent-written tests.
→