Judge the Proxy, Not the Wall Clock
I’m regularly optimizing the software I build, which means I’m also regularly looking for benchmarks that can tell me whether an improvement is actually meaningful.
When I’m working with coding agents, though, I’ve noticed that it's easy to be surprisingly sloppy about this. The easy default is often wall-clock time: make a change, run something before and after, and see whether it got faster.
The problem is that wall time can be noisy and, more importantly, may not actually measure the mechanism I’m trying to improve.
I’ve started using a simple prompt and skill that force the agent to identify the right proxy before evaluating an optimization. The idea is to first state what the change is actually supposed to affect, then choose a count or ratio that measures that mechanism directly using data available in logs, labels, or annotations.
That makes it much harder to declare victory because a convenient benchmark happened to move in the right direction for unrelated reasons.
Here’s the prompt:
Judge the changes with a proxy, not wall time. First name the mechanism the change is supposed to affect. Then pick one count or ratio that measures that mechanism directly, one you can read from logs, labels or annotations. Report its source, the sample size and a before/after, plus a control that must return non-zero. Treat wall time as context only, and say “insufficient sample” if n is small.
It’s a small change, but it has made my agent-driven performance work more rigorous: define what should change, measure that thing directly, and only then look at whether the software also got faster.
I also turned this into a skill called Proxy-first evaluation. It makes agents measure the thing an optimization actually changes rather than reaching for wall-clock time or another convenient but potentially misleading benchmark. It also requires the agent to identify the source of the measurement, its detection floor and sample size, and to pair zero results with a control that proves the instrument is actually working.
Generous-Corp