Data & AI
Evaluate the model
Vibes are not a metric. Build the eval, measure the misbehavior, re-run when anything changes.
Requires The Harness IV
The Harness taught measuring your machine in general; here the subject is the models you build and buy. An evaluation is a fixed task set, a held-out grader, and a number you can re-earn on demand. Without one, every model change is vibes: the demo looked good, the stakeholder nodded, the regression shipped. With one, an upgrade is a diff you can read.
Design the eval like the adversary it is. The grader lives outside the thing it grades, or the thing learns the grader instead of the task. The test set stays held out, or the number measures memory. And the conditions stay hermetic: a measurement taken inside a warm, context-rich session tells you about that session, not about the model. Clean rooms exist because instruments contaminate.
How to practice it
Freeze the task set.
A benchmark that drifts with the model measures nothing. Version it, and re-run the old set on every new candidate.
Hold out the grader.
Never let the graded system see, choose, or score its own test. Judges with a stake acquit.
Hermetic or it lies.
Fresh session, no shared context, no helpful warm-up. Contaminated conditions flatter exactly when you need truth.
Numbers over anecdotes.
One good demo is a story; twenty scored runs are a measurement. Decide upgrades on the second.