Frame 04Open sourceLLM evaluation2026
A multi-agent LLM judge built to fail loudly
Project D3 is my Python reimplementation of Debate, Deliberate, Decide (arXiv:2410.04663): advocate agents argue for two candidate answers, a judge scores the defenses, and a five-persona jury votes. The method is the paper's, not mine. The time went into the part a paper does not have to cover: what the pipeline does when a model call fails, when a juror answers off-format, or when the winner is really just the answer shown first.
- Repo
- Project_D3
- Paper
- arXiv:2410.04663, Debate, Deliberate, Decide
- Stack
- Python, OpenAI chat completions, tiktoken, ThreadPoolExecutor, argparse CLI
- Scope
- Sole author. A reimplementation of a published method, not a fork and not a new method.
- Both D3 protocols as a CLI (MORE: one round, three advocates per side; SAMRE: one advocate per side, up to five rounds) with a batch runner over MT-Bench and AUTO-J.
- On 20 MT-Bench pairs, MORE matched the human label 68.8% of the time, Cohen's kappa 0.375, at about 19,051 tokens per evaluation. README figures from one run; no saved report in the repo.
- No tests, n of 5 to 20, three of four rows at kappa 0.000. The failure handling is the work. The agreement numbers do not prove much.
What could go wrong, and how we would know
One MORE evaluation at the defaults is 14 model calls: six advocates, two aggregation calls, one judge, five jurors. Any of them can time out, come back empty, or come back as text the parser cannot read. The quiet failure is the dangerous one. An advocate's error string becomes a "defense" the judge dutifully scores. A judge with no parseable score tuple gets treated as a tie. A juror who wrote "Score for Answer 1: 250" gets counted.
The jury has its own trap. Five jurors, three options: Answer 1, Answer 2, Tie. A vote of 2, 1, 1 with one vote unreadable looks like a win for Answer 1 by plurality. It is not a majority of anything.
And the whole thing can be right for the wrong reason. An LLM judge can favor a position rather than an answer; the paper measures this by swapping the answers, and so does the runner. Remap the swapped winner back to content labels before comparing, and a judge that always picks position 1 looks perfectly consistent.
Every agent call returns a success flag next to its text. MORE aborts with an explicit error when a whole side of advocates fails and continues on partial defenses otherwise; aggregation and judge failures abort too. SAMRE stops with a named stop_reason (convergence, budget, max_rounds, advocate_failure, judge_failure) and skips the jury when zero rounds completed. Juror scores outside 1 to 100 are discarded, a missing Verdict line is inferred from the scores, and an unreadable vote is retried up to two more times. A verdict needs more than half of the valid votes; anything less goes to the judge's cumulative score. Position bias is computed on raw positional winners, with a comment in the runner saying why.
What I built
The pipeline follows the paper's two algorithms. In MORE, three advocates per side write arguments in parallel on a ThreadPoolExecutor, one call per side merges them into a single defense, and the judge scores the pair once. In SAMRE, one advocate per side argues for up to five rounds; each round the judge returns scores plus feedback addressed to each advocate separately, and the loop ends when the score gap stops moving (threshold 0.05) or the token budget runs out. Either way a jury of five personas from the paper's Appendix D.3 reads an anonymized transcript and votes.
The judge scores six criteria at 1 to 20 each and must end with a FINAL_TUPLE: (s1, s2) line; the parser tries that exact line first and falls back to the last tuple in the text. Jurors give each answer 1 to 100 plus a Verdict line. All user content in every prompt is wrapped in <<<...>>> delimiters, so an answer that contains instructions is at least marked as data.
The batch runner loads MT-Bench, AUTO-J or AlignBench into one schema and reports agreement rate, Cohen's kappa and token cost. With --measure-bias it runs every sample twice, once swapped, and records both the raw and the remapped swapped winner.
| Chose | Over | Because | Cost |
|---|---|---|---|
| Strict majority (more than half of valid votes) | Plurality | With five jurors and a Tie option, 2-1-1 plus a dropped vote is not a decision | More verdicts fall through to the judge tie-break |
| Judge cumulative score as tie-break | Returning a tie | The paper's rule; keeps a usable winner for the agreement stats | A tie-broken verdict counts the same as a 5-0 one |
| Retry an unreadable juror, up to two more attempts | Dropping the vote | One dropped vote moves the majority line from 3 of 5 to 3 of 4 | Up to three times the juror tokens on a bad day |
| Raw positional winners for the bias metric | Remapped winners | After remapping, "position 1 always wins" reads as a consistent verdict | Two fields per sample record, easy to confuse |
| Two-class kappa, ties excluded from both sides | Three-class kappa | A human tie carries no preference to agree with; the comparison is Answer 1 against Answer 2 | A judge that ties often loses agreement but drops out of kappa |
Try it
This ports aggregate_votes from agents/jury.py and cohens_kappa from evaluation/metrics.py. Set five votes and the judge's cumulative scores; the readout shows the winner, majority_reached and tie_broken. The 2-1-1 preset is the edge case the docstring calls out.
Verification
The README reports four runs, all with gpt-5.4-nano as the backbone for every agent role. They are small.
| Dataset | Protocol | n | Agreement | Kappa | Mean tokens per eval |
|---|---|---|---|---|---|
| MT-Bench | MORE | 20 | 68.8% | 0.375 | ~19,051 |
| MT-Bench | MORE | 5 | 60.0% | 0.000 | ~15,911 |
| MT-Bench | SAMRE | 5 | 60.0% | 0.000 | ~15,424 |
| AUTO-J | SAMRE | 5 | 80.0% | 0.000 | ~14,648 |
Read the two metric columns together: agreement counts a D3 tie against a clear human preference as a miss, while kappa drops any pair where either side is a tie. There are no unit tests in the repo and no saved run report; the numbers are what the README says, from one run each.
- 68.8%agreement with human labels, MT-Bench, MORE, n = 20
- 0.375Cohen's kappa on the same run
- ~19,051mean tokens per evaluation
What I found
Kappa 0.000 is a signature, not rounding. With five samples and the repo's own formula, kappa is exactly zero next to 60% or 80% agreement only when one side never changes its label: either the judge gave the same verdict on every counted pair, or the human labels on those pairs were all the same. Observed agreement then equals chance agreement by construction. The data says which. --max-samples 5 takes the first five samples in file order, and the first five lines of both mt_bench_pairwise.jsonl and autoj_pairwise.jsonl carry the same human label, Answer 2. On those rows kappa can only be 0.000 or 1.000, and 1.000 needs a perfect score, so the three n=5 rows say nothing about the judge either way. The 20-sample row is the only one with both human labels in it (16 non-tie labels, 11 matched). What would have hidden it: reporting agreement alone. 80.0% on AUTO-J reads well until the 0.000 beside it.
Outcome, and what it does not prove
What the runs show: fair agreement (kappa 0.375) on 20 MT-Bench pairs with a small model, and nothing statistically meaningful below that. What they do not prove: that D3 beats a single-call judge (never compared), that the jury adds anything over the judge alone (never ablated), or that the pipeline is position-neutral (the flag exists; no bias run is reported). AlignBench has a loader and 683 samples of data and no result.
At the default 4,096-token budget the SAMRE loop likely stops after round one: the budget check runs after each round, and two advocate calls plus a judge call probably exceed it by themselves. I have not verified that against a logged run, so multi-round debate at the defaults should be treated as untested.
What I would do differently
- Tests before benchmarks.
aggregate_votes,cohens_kappa,position_biasand both parsers are pure functions; the 2-1-1-tie case and the out-of-range juror are short tests that do not exist. - Sample past the first N lines of the file, run
--measure-biason the same 20 samples before quoting any agreement number, and print the bias rate beside it. - Raise the SAMRE default budget or make it per round, and report the distribution of
rounds_completed, so a debate that never debates is visible. - Run the five jurors in parallel like the advocates; deliberation is five model calls in a row.
- A reimplementation of arXiv:2410.04663 (upstream: abirharrasse/D3-Judge). The protocols and personas are the paper's and the advocate prompts come from its Appendix F; the juror prompt (the paper prints none), the failure handling and the metrics code are mine.
- No tests and no CI anywhere in the repo.
- Sample sizes are 5 to 20; three of four rows report kappa 0.000, and the n=5 rows are the first five lines of each file, all with the same human label; the README numbers are single runs with no saved report checked in.
- AlignBench has a loader and data only. No reported run, so "benchmarked on AlignBench" would overstate it.
- The default 4,096-token budget likely ends SAMRE after round one. Unverified; inferred from the config and the README's cost table.
- Jurors vote sequentially even though the advocates run in parallel.
Sources README agents/jury.py evaluation/metrics.py evaluation/runner.py protocols/samre.py config.py data/ Scope: I (sole author) Verified 3 Sep 2026