Every few weeks a lab ships something and the leaderboards reshuffle. Swurl will change its defaults considerably less often than the leaderboards change, and will frequently disagree with the ranking. Here is the evaluation we intend to run, and why it is built this way.
Benchmarks measure the wrong distribution
Public benchmarks are built to discriminate between strong models, which means they are weighted towards hard problems. Real traffic is not. Our measured task mix is roughly 55% work that any competent model handles, 20% coding, 15% long-context analysis, and 10% genuinely hard reasoning.
A model that wins a reasoning benchmark by four points may be irrelevant to 90% of what a workspace actually sends. Meanwhile a two-hundred-millisecond difference in time to first token, which no benchmark reports, changes how people feel about the product every single message.
What we will run instead
- A replay set of 4,000 real prompts, sampled to match production task mix, stripped of customer content and rebuilt as synthetic equivalents. This is the part that matters most and the part no public benchmark can give us.
- Pairwise preference against the incumbent. Not absolute scores. For each prompt, which answer would a competent reviewer prefer, judged blind. A new model has to win clearly, not marginally.
- Failure-mode probes. Specific tests for the things that make a model unusable in production regardless of average quality: instruction drift over long threads, silent truncation, fabricated citations, ignoring an explicit format request.
- Latency and variance under load, measured at the ninety-fifth percentile over a week, not the median over an hour. A model with a good median and a terrible tail is worse in practice than a slightly slower, steadier one.
- Cost per accepted answer, not cost per token. A cheaper model that needs two attempts is not cheaper.
Where we expect to disagree with the leaderboards
Based on how these failure modes usually play out, the recurring patterns should be:
- Verbosity inflates benchmark scores and irritates real users. Several models that grade well produce answers twice as long as necessary, which costs more and reads worse.
- Instruction adherence is underweighted. A model that is slightly weaker but reliably respects "answer in three bullets" is more useful in a product than a stronger one that does not.
- Tail latency is invisible to benchmarks and extremely visible to users.
- Long-context claims degrade quietly. A stated two-million-token window rarely means usable recall across two million tokens. We test retrieval at 25%, 50% and 90% depth and the drop-off is often severe well before the limit.
Why the default keeps moving anyway
Because prices move as much as capabilities do. A model that was not the cost-effective default in May becomes it in July when its price halves, with no change in the model itself. Roughly half of our default changes over the past year were driven by pricing rather than quality.
This is also the argument for not buying a single subscription. Whichever lab is the right default today is unlikely to be the right default in six months, and being able to move without a procurement cycle is worth more than picking correctly the first time.