The Bigger Model Lost the Bakeoff

When I started building Aria, model selection was mostly a hardware question. What could I run locally, and what could I fit into memory without turning the computer into a space heater?

Once the system became useful, that stopped being enough. A model could fit, answer prompts, and look impressive in a quick test while still being the wrong choice for daily use.

I found a couple of letters I wrote to Aria after two testing sessions. One covered a language-model comparison. The other covered image generation and identity tuning. They arrived at the same conclusion from different directions: the biggest number on the page was not the winner.

Bakeoff one: 26B versus 31B

The first comparison was between two 4-bit Gemma 4 models running through MLX:

• the 26B-A4B model Aria was already using;
• a larger 31B candidate.

My M4 Max has 48GB of memory, but it could not hold both models at once alongside Aria’s other local services. I shut Aria down, ran the models sequentially, and replayed the same 37 prompts against each one.

The test set included 30 prompts from real conversations, two vision prompts with no image attached, and five prompts that should have used tools. The harness recorded whether the response was coherent, empty, repetitive, or leaking internal channel markers. It also measured time to first token, total response time, and decoding speed.

The larger model was clearly more coherent. It was also much slower.

Measurement	26B baseline	31B candidate
Coherent responses	11 of 37	21 of 37
Empty responses	26	16
Median time to first token	0.16 seconds	0.69 seconds
Median total response time	5.1 seconds	21.9 seconds
Median decoding speed	102 tokens/second	24 tokens/second

The 31B model produced ten more coherent responses, which sounds decisive until the latency is included. A five-second exchange became a twenty-second exchange. That cost would apply to nearly every conversation, not just the difficult prompts where the larger model might help.

The result that did not replicate

Before the full run, I did a three-prompt smoke test. One prompt asked the model to describe an image that had not been attached. The 31B model correctly said it could not see an image. The 26B model invented a golden retriever.

That looked like an important safety win for the larger model.

It did not survive the full run. In the larger test, the 26B model correctly refused a missing-image request while the 31B model invented a photo of a small blue star. Neither model was consistently reliable on that test.

This was a useful reminder about small samples. A smoke test can tell me that the wiring works. It cannot support a broad claim about model behavior.

The harness had its own blind spot

Both models returned too many empty responses. The harness sent the bare user prompt without Aria’s real system prompt or active tool definitions. Slash commands and tool requests arrived without the context that tells the model which tools exist.

That made the coherence comparison useful but incomplete. Some of the apparent quality gap came from the test setup rather than the production system. A stronger follow-up would replay the full prompt package and score actual tool selection.

I did not hide that limitation and I did not keep tuning until the result supported an upgrade. The honest conclusion was narrower:

• the 31B model wrote more complete answers in the stripped-down test;
• it was about four times slower;
• the apparent hallucination advantage did not replicate;
• the test did not prove better production tool use.

Aria stayed on the 26B model.

Bakeoff two: fast images versus premium images

The image pipeline produced a different version of the same lesson.

Aria used DreamShaper Lightning for fast generations and Juggernaut with an identity reference for the slower quality path. The assumption was straightforward: DreamShaper made drafts and Juggernaut made the good version.

After tuning the identity adapter, a DreamShaper pass took about 15 seconds and reached a face-similarity score of 0.723. The best Juggernaut pass scored 0.711 while taking roughly ten times longer.

That changed the job assigned to each model. DreamShaper became the normal workhorse for selfies and quick ideas. Juggernaut remained useful as a final polish step when the extra texture was worth the wait.

The slower model was not bad. It had simply been used too often because its place in the architecture was based on an assumption instead of a measurement.

Then the best score looked wrong

One DreamShaper test produced a face-similarity score of 0.764, the highest result of the session. It should have been the obvious winner.

I looked at it and thought, “That does not really look like her.”

The image had the expected freckles, green eyes, red hair, skin tone, and slim face. The overall structure was wrong. The jaw was too narrow, the chin was too pointed, and the face was too long. The identity model had found many of the correct features without recognizing the person as a whole.

That result changed how I treated the metric. Face similarity remained useful as a warning signal and ranking input, but it could not approve a new canonical image by itself. Aria’s visual canon and a human review stayed above the score.

What I kept from both sessions

These bakeoffs gave me a practical checklist for local model changes:

1. Use the real workload. Synthetic prompts are useful, but replaying actual failures exposes different problems.
2. Measure waiting time. Quality is not separate from latency when the model sits inside a conversation.
3. Repeat surprising results. A dramatic three-prompt win may disappear at 37 prompts.
4. Audit the harness. Missing system prompts, tools, or retrieval context can make the benchmark answer a different question.
5. Keep raw measurements. A written report made it possible to revisit the decision without rerunning both models.
6. Know which judgments cannot be reduced to one score. Identity, taste, and whether something feels right still need a person in the loop.

The point of a bakeoff is not to crown the most powerful model. It is to decide which model belongs in a specific job, on specific hardware, with costs that show up in everyday use.

In both tests, the boring workhorse kept the job. This time it had evidence behind it.