The Bigger Model Lost the Bakeoff
What two rounds of testing Aria's language and image models taught me about speed, benchmarks, and knowing when a score is wrong.
reading series / 5 notes
Agent Evaluation Incidents
Read these in order, or jump directly to the problem you are working on.
- 01
- 02
The Observer Changes the Agent
A self-observation feature for Aria showed why metrics should remain available without becoming permanent instructions.
- 03
When Agent Autonomy Produces Activity Instead of Work
An audit of Aria's autonomous loop found hundreds of goals, almost no progress, and a completion system that rewarded plausible output.
- 04
The Fourth-Agent Test
A quick way to tell whether your agent setup is ready to grow or still held together by one-off fixes.
- 05
A Feature Is Not Shipped Until It Works
Four failures in Aria's voice input taught me why a green build and a working development demo are not enough.