Meta's AI writes a test for each risky code change before it merges, and caught 74 serious privacy bugs this year
On 29th Oct'26 Meta's engineering team described Catching JiTTests, a system that uses language models to write tests for individual code changes before they land, rather than having a model read the code and judge it. Overnight, on spare capacity, it picks the previous day's changes scored as risky and writes tests designed to expose a difference in behaviour, either by treating the change itself as a possible bug or by guessing its intent and the ways it could go wrong. Failing tests are screened by several models acting as judges and by a fixed set of rules, and only the highest-ranked findings reach an engineer, who confirms whether the change in behaviour was intended. Meta says 589 engineers have engaged with it, 64% of those contacted, and those who did rated 60% of its findings as real. It caught 74 serious privacy bugs before they reached production in the first half of 2026, and Meta credits it with a 20% fall in serious privacy incidents over that period. Meta does not say what it costs to run, and the figures are Meta's own.
What it actually means
Most teams adding AI to code review ask a model to read the change and comment, which produces opinions engineers learn to skim past. Meta's design asks for evidence instead, because a failing test shows a real change in behaviour and the engineer's only job is to say whether that change was meant. On 29th Oct'26 we covered research showing human reviewers drift towards whatever an agent argues for, and this design works the other way round, since the human judges an observed result rather than an argument.
The parts worth copying are the cheap ones. Run it overnight on yesterday's riskiest changes rather than on everything, and put a fixed rule filter beside the model judges so a noisy model cannot flood your engineers. Even after that filtering, engineers rejected about four findings in ten, so treat it as a second pair of eyes on risky changes and not as a gate that blocks a merge.
Ask the model for a failing test, not an opinion, and the human only has to judge what happened.
74serious privacy bugs Meta's generated tests caught before production in the first half of 2026
Source: Meta Engineering · Saturday, 31 October 2026