An evaluator without a rubric is a preference

Calibrated rubrics, and why calibration is the whole job

The task sounds trivial: look at what a frontier image model produced and say how good it is. The trouble is that "good" is not a measurement. Two raters applying their own sense of quality produce two different datasets. Two raters applying the same rubric produce one, and that one is what an RLHF pipeline can actually learn from.

The rubric named the axes, and they were the ones that resist a quick glance: rendering artifacts that only show at certain scales, and anatomical consistency, the hands, joints and symmetry that a first impression reads straight past. Calibrated, here, means the criteria had been tuned so that independent raters applying them converge instead of drift. My job was to hold the line the rubric drew, image after image, including on the days when my own taste disagreed with it.

That is the direct ancestor of how I build automated evaluation now. A golden set with explicit criteria is a rubric that a machine can be held to. Measuring an LLM judge's position bias, as I did in Project D3, is checking whether the evaluator is calibrated instead of assuming it. The rubric verifier and the golden evaluation set at Verita AI are the same idea with the human rater replaced by a model, and the same question asked of it: can it say why.

Reviewing machine-learning code

The second strand was reviewing Kaggle machine-learning codebases for algorithm selection and implementation quality. A leaderboard score tells you a notebook did well once. It does not tell you whether the algorithm fit the problem, or whether the implementation did what the write-up claimed it did.

Those two questions were the review. Is this the right family of method for this data, and does the code actually implement that method, with the evaluation set up so the number means what it says. It is the same habit as the rubric work, pointed at code instead of images: the claim has to survive being checked against a stated criterion.

Star Fellow

I was selected as a Star Fellow for a program evaluating model reasoning against frontier research publications. The shape of the work: put the reasoning in a research paper in front of a model and check where it tracks the argument and where it does not. The program's materials are not public, so I can describe its shape here and not its results.