The Harness
Evaluate your AI
Trust is measured, not felt. Find where it fails before it matters, and measure again when the machine changes.
You have opinions about how reliable your machine is. Opinions are vibes; trust should be a measurement. Give it the same task eight times in a clean room and count what comes back: how often it falls into the trap, which steps it skips, where confidence outruns correctness. The numbers are usually a surprise in both directions, better than feared on some skills, worse than assumed on exactly the ones you stopped checking.
Two calibrations deserve standing attention. Its estimates run long: measure the ratio on your own tasks and plan with the corrected number, not the quoted one. And every measurement expires when the model changes; the machine you calibrated last quarter is not the machine you are running today. Re-measure on upgrade, the way you would re-baseline any instrument.
How to practice it
Test where it matters, before it matters.
Build one small repeatable task per skill you depend on. Run it cold, several times. Count, don't recall.
Keep the measurement honest.
A test the machine can see coming, or one graded by the machine that took it, measures nothing. Fresh setting, held-out grading.
Calibrate the estimates.
Track quoted-versus-actual until you know the multiplier. One operator's measured ratio: three to four.
Re-baseline on every upgrade.
New model, new numbers, in both directions. Yesterday's known weakness may be fixed; yesterday's safe assumption may not be.