Google AI Research Releases Agile Deliberation Framework for Gray-Area Image Classifiers

Google Research has released Agile Deliberation, a human-in-the-loop framework for building image classifiers around subjective concepts. Think ‘healthy food’, ‘inspiring artwork’ or clickbait in content moderation. Most tooling assumes users already know the concept boundary. Agile Deliberation assumes they do not, and helps them find it.

What is Agile Deliberation?

Agile Deliberation is a Google Research’s framework that turns a vague concept into a structured written definition. That definition does two jobs. It is a human-readable policy, and it is the text prompt for a VLM classifier. Each round keeps the candidate definition with the best F1 on the user’s labels.

The design came from interviews with a group of content moderation experts. Experts scoped a concept first, then refined it on borderline images. Finding those images was the slow part.

How does concept scoping work?

Stage 1 breaks the concept into at most 3 unit concepts, using prompt chaining with Gemini 2.5 Pro. Each unit gets candidate positive and negative subconcepts, shown with retrieved images. For healthy food, positives might be salads or grain bowls. Negatives might be processed food or raw, unprepared ingredients. The user keeps, flips or drops each one.

How does concept iteration find borderline images?

Stage 2 skips classic uncertainty sampling. The classifier is a prompted generative VLM, so its confidence scores are often uncalibrated. Instead, the system hunts for images that are borderline in meaning.

The pipeline runs 4 steps:

Gemini 2.5 Flash writes borderline queries, such as ‘salads with heavy mayo dressings’. Each query retrieves 50 to 100 images.

Images are de-duplicated, encoded, and grouped into clusters with dictionary learning.

A UCB multi-armed bandit picks the next cluster. It favors clusters where users corrected the model or left rich feedback.

A VLM summarizes each image in 1 sentence. The user sees at least 5 images that share 1 ambiguity.

Users label each image and add short notes on disagreements. An LLM turns those notes into candidate definitions. A greedy update keeps the best-scoring one.

How was it evaluated?

Subjective concepts have no fixed ground truth, so Google team skipped static benchmarks. It ran 18 live sessions of about 90 minutes each. 9 participants without professional moderation experience did 2 sessions each. One session used Agile Deliberation, the other used manual deliberation. Each participant spent about 45 minutes defining a concept, then labeled 200 held-out test images.

The two concepts were ‘paid to play’ clickbait and ‘healthy food’. Retrieval ran over a proprietary 100M-image dataset and WebLI, respectively.

What do the benchmarks show?

Source: Table 1 of the CVPR 2026 paper. F1 averaged over participants.

How does it compare with other subjective classifier methods?

How can developers run it?

The repo ships a Demo.ipynb that walks through the healthy food example. It needs a Gemini API key and defaults to gemini-2.5-pro and gemini-2.5-flash. A setup script builds a FAISS index over LAION-400M CLIP embeddings. The default 2 shards take about 260 MB. All 100 shards take about 13 GB.

Check out the paper, arXiv version, GitHub repo and Google Research blog. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us