Seven worked examples, graded.

These are the engagements we are unusual in being able to take: the model runs on the client’s own hardware and nothing reaches a third party. That used to cost consistency. On the current model it does not — the local column matches the better of the two hosted models question for question, and answers faster than either.

Most organisations are here, and many do not realise it. The data carries no residency constraint, so an approved hosted model is appropriate — nothing to operate, no hardware to buy, quicker to stand up. We will say so rather than sell you infrastructure you do not need.

Most organisations do not need a private model, and we will say so. If your policy already permits an approved AI vendor, a hosted frontier model is the least work: no GPUs to buy, nothing to operate, and the accuracy shown above. That remains our default for most clients. What it is not, on these questions, is the fastest.

What we will not do is pretend that choice is free of consequence. In that configuration your prompts, and the query results the agent reads, are sent to the model provider under your agreement with them. For a great many workloads that is entirely acceptable and already covered by an existing vendor review. For some it is not, and that is the case we are unusual in being able to handle.

Hosted — our default

Nothing to operate and no hardware to buy, priced per token. The most accurate single configuration measured here, though only just. Data reaches the model provider. Right for most clients most of the time.

On your hardware

For data that genuinely cannot leave. Nothing reaches a third party and there is no per-token cost once the hardware exists. Accuracy depends heavily on which open model you run: the local column is a 36B mixture-of-experts on one machine, and moving to it from a 30B coder model closed the gap to the hosted models entirely. So “a bigger model would help” is not the lesson — which model, and how it is tuned, mattered far more than size.

Mixed

Usually the right answer once a deployment matures: restricted data local, everything else through an approved API, decided per use case rather than per vendor.

Switching later

One line of configuration. You are not choosing once and living with it, and you are not locked to us either way.

One result is worth dwelling on, because it reverses the obvious assumption. Earlier measurements had both hosted models more accurate than the local one and slower than it, and we put the latency down to round trips rather than to the models: this is an agentic loop that discovers the schema, describes tables and runs a query across several calls, and when the model is hosted every one of those crosses the internet. Changing the local model tested that reading. The current one matches the better hosted model question for question and answers in about 230s against their 391 and 403. The round trips were the cost. “Hosted is fast, local is private” was always too simple — on these questions, local is now the faster of the two.

OpenAI 20–21/21 × 3 · 374–411s DeepSeek 21/21 × 3 · 329–591s Local 20–21/21 × 4 · 215–242s

Results vary by model, by hardware and by run. Each model here was graded repeatedly, and the figures are ranges and medians across those runs rather than a single measurement — but they were all taken on one date, and they are not a guarantee. We have seen the same local model answer a question correctly in one run and incorrectly in the next, which is exactly why the table counts runs instead of showing one verdict. If a specific model matters to you, the honest answer is to run these graded questions against it rather than trust a table on a website — and we will do that with you, on your schema.