Meet Meta’s JiTTests: LLM-Generated Tests Designed to Fail Before Risky Code Lands
Meta has moved Catching Just-in-Time Tests (JiTTests) from research prototype into its production infrastructure. Meta JiTTests is an LLM-based system that writes executable tests designed to fail on a risky code change. A failing test is concrete evidence that a diff changed behavior. Meta team reports that JiTTests caught 74 privacy SEVs at the diff stage in the first half of 2026. The method is described in the FSE 2026 research paper ‘Just-in-Time Catching Test Generation at Meta‘.
What are Catching JiTTests?
Catching JiTTests are LLM-generated tests that should fail when a diff introduces a bug. Traditional hardening tests pass when written and guard against future regressions. Catching tests target the change under review, before it lands. Meta formalized the details in its 2025 paper ‘Harden and Catch’.
Most AI bug tools act like faster human reviewers. They read the diff and leave comments. JiTTests instead execute code and show a concrete behavioral difference. Meta team concedes this takes more computational steps. It argues the approach catches bug classes that LLM or agent review can miss.
How does the JiTTests pipeline work?
The production pipeline has 5 stages:
Targeting: Runs overnight on spare capacity. It selects the previous day’s risky diffs, using a mechanism similar to Meta’s Diff Risk Score.
Test generation: 2 diff-aware workflows produce candidate tests (explained below).
LLM-as-Judge ensemble: Several LLMs score each failing test using the test code, failure trace, diff and inferred intent.
RubFake: The Rule-Based False-Positive Killing Environment adds deterministic checks. Its rules flag patterns like reflection, broken mocks and not-implemented exceptions.
Human-on-the-loop: The system decides whether to contact an engineer, based on the combined ranking.
Dodgy diff vs intent-aware: what is the difference?
The dodgy diff workflow assumes every diff is buggy. It treats the diff as a mutant of its parent and generates tests that tell them apart.
The intent-aware workflow first infers what the diff should do from its code, title and summary. It lists risks, builds a mutant for each, and writes tests to kill them. Tests that then fail on the real diff become ‘weak catches.’
What do the benchmarks show?
The research team analyzes 22,126 generated tests. Diff-aware methods produced 4x more candidate catches than hardening tests. They produced 20x more than tests that failed by coincidence.
The intent-aware workflow produced weak catches in 6.4% of its tests and on 7.9% of diffs. Dodgy diff scored 2.5% and 4.0%. Meta’s earlier ACH mutation-guided hardening scored 0.8% and 1.6%. The rule-based and LLM assessors cut human review load by 70%.
*Figures from Table 1 of the JiTTests paper (self-reported, Meta codebase). TestGen-LLM row uses the “need not add coverage” setting. A weak catch is a candidate failure, not a confirmed bug.
What changed at full scale?
The initial deployment was modest. Meta reported 41 candidate catches to engineers. 8 were confirmed true positives, a 19.5% rate. 4 of those would have caused serious failures in production.
The October 2026 engineering post reports much stronger production numbers:
589 engineers engaged with JiTTests, 64% of those contacted.
Engineers reported a 60% true-positive rate.
74 privacy SEVs were caught at the diff stage in the first half of 2026.
Overall privacy SEV occurrence fell 20% in the same period.
What does human-on-the-loop mean?
Human-in-the-loop means a person reviews every output. Human-on-the-loop means engineers set criteria and monitor results in aggregate. They intervene at the policy level, not on each test. Meta team rarely need to read test code. They only confirm whether an observed behavior change was intended.
Can developers use JiTTests?
The answer is No. JiTTests is internal Meta infrastructure. Meta has not released code, models or an API. Teams can still copy the pattern: diff-aware mutation-guided test generation, an LLM judge ensemble, deterministic filters and aggregate human oversight.
Check out the engineering post, the FSE 2026 paper and Meta’s announcement on X. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.
Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Originally published by marktechpost.com →