We tested our model router on 221 prompts. Here’s what broke.
· 6 min read
Apier’s Auto picks a model for every message: a fast, cheap one for “thanks!”, a frontier model for a proof. If it picks well, you get better answers and your credits last longer. If it picks badly, you get a weak answer to a hard question, or pay top-model prices for small talk.
So we tested it properly, and it was worse than we thought.
The test
We wrote 125 realistic prompts: homework, stack traces, proofs, lease questions, poems, product comparisons, and everyday messages. Each got a label for its field (code, math, law, creative writing and ten more) and how hard it is (easy, medium or hard). Then we ran every prompt through Auto, with every model available, and compared.
We also wrote a second set of prompts and didn’t look at them while fixing anything. Rules always look great on the examples they were built from; the unseen set tells you how they do on real messages.
What we found
- The field was right 68% of the time on the prompts we knew, and 44% on the unseen ones. “Why is the sky blue?”, “What do mitochondria do?” and “What is an index fund?” were all treated as plain chat.
- Poems went to the humanities. The rules checked for literature before creative writing, and both lists contained “poem”.
- Everyday words set off traps. “My cell phone battery dies by noon” was biology (cells). “How do I solve a fight with my roommate?” was math (solve). “Plan a 3-5 day trip” was math too: 3 minus 5. “What’s a good java place downtown?” was code.
- One model got 38% of all messages. It had top scores in several fields and the lowest price in its tier, so it won nearly every medium-difficulty message, including plain chat it isn’t especially good at.
- A cheap top-tier model won hard tasks it shouldn’t. Its scores had been set as high as the leaders’, and price broke the tie. Independent tests put it well behind them on hard coding.
What we changed
Scoring instead of first match. Each field now has signal phrases with weights. “Derivative” or “haiku” decides on its own; “compare” only counts with other signals. The field with the most points wins, so “write a poem” is creative because creative outscores humanities on it.
Trap guards. “Cell” doesn’t count before “phone”, “solve” needs something to solve, a dash between numbers isn’t subtraction, and “java” before “place” is coffee.
Needs. Auto now notices whether a message needs creativity, reasoning or a clear explanation, and favours models strong at that.
Scores rebuilt from research. We went through independent indexes, leaderboards and reviews, and rescored every model in every field. Half points mark where evidence separates two models without a full point.
One file for every setting. How much price counts, how much creativity counts, which fields never go to a cheap model (law), and which models Auto never picks. Changing behaviour now means changing a number, then rerunning the tests.
The results
- On the prompts we tuned on: field right 99% of the time, difficulty 95%.
- On the first unseen set: field 44% → 76%, difficulty 82% → 88%.
- On a second unseen set, written after all of the above: field 78%, difficulty 74%. That’s our honest number.
- No message, in any run, was judged easy when it was hard, or hard when it was easy.
- The model that took 38% of messages now gets about 5%, for medium math, where it’s strong and cheap.
What rules can’t do
Every new batch of prompts brought words the rules had never seen: “TCP and UDP”, “Cleopatra”, “fever”, “invent a holiday”. You can keep adding words, but the unseen score stays in the high 70s. The misses are mostly mild: a short question read as easy, so it goes to a fast model instead of an everyday one.
The fix is a small, fast model reading each message, with the rules as a backup. That’s next, and we’ll publish the numbers the same way, including the misses.
Why publish the misses?
Because “we pick the best model for you” is easy to say and hard to check. How Auto works shows every score, every test, and every prompt Auto still gets wrong, computed from the same code that routes your messages. If you find a message it handles badly, tell us at support@apier.ai: it goes into the test set.