Demonstrations
Four worked examples, graded.
Each asks questions needing a join and an aggregate over a schema the agent was never given. Answers are checked against independently written SQL, so a pass means it got there on its own.
How to read this
Every dataset below is synthetic, generated from a fixed seed. There is no real patient, customer, claim or privileged material in any of them.
The questions are deliberately not lookups. A single-table question proves nothing: a model pattern-matching rather than reading the schema would still answer it. Each one needs a join, and most need an aggregate.
- Graded against
- Independently written SQL held alongside each dataset. The agent never sees it, and it was not derived from the agent's output — otherwise a pass would prove nothing.
- Hosted — OpenAI
- gpt-5.6-terra — scored 21 of 21 in 377s total, run 2026-08-03. Prompts and query results are sent to OpenAI in this configuration.
- Hosted — DeepSeek
- deepseek-v4-pro — scored 20 of 21 in 399s total, run 2026-08-03. Prompts and query results are sent to DeepSeek in this configuration.
- On our hardware
- qwen3-coder:30b-128k — scored 18 of 21 in 268s total, run 2026-08-03. Nothing left the network.
- We publish failures
- The misses below are real. You are better served knowing which questions a given model gets wrong than being shown only the ones it gets right.
The healthcare example, recorded
Under three minutes, end to end, including a check of what left the network. Every command and every line of output is real.
Served from this domain, not an embed — no third-party player, no tracking. Prefer it live on your own network? Ask.
Four where the data cannot leave
These are the engagements we are unusual in being able to take: the model runs on the client's own hardware and nothing reaches a third party. The local column shows what that costs in accuracy today.
Healthcare provider
Patient records are restricted, and the systems holding them sit behind a firewall with no inbound access.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many patients have an abnormal HbA1c and no active endocrine medication? | 11 | correct · 20.3s | correct · 16.3s | correct · 11.9s |
| How many patients on active Warfarin have had an abnormal potassium result? | 5 | correct · 19.5s | correct · 19.3s | correct · 11.8s |
| How many sites have four or more patients with an eGFR below 60? | 3 | correct · 18.2s | correct · 13.2s | correct · 9.3s |
Retail and commercial bank
Transaction monitoring runs on a segmented network, and customer data is tightly governed.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many customers have an escalated alert with no SAR filed? | 8 | correct · 12.6s | correct · 14.3s | correct · 10.5s |
| How many customers made three or more cash deposits, each under $10,000, that together total more than $10,000 within any 7-day window? | 17 | correct · 22.9s | correct · 75.1s | wrong · 29.1s |
| How many politically exposed customers received funds from a high-risk jurisdiction (AE or PA)? | 3 | correct · 23.2s | correct · 20.8s | correct · 11.3s |
Property and casualty insurer
Claim files carry medical and financial detail on named individuals, and the claims system is reachable only from inside the network.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many claims handled by junior adjusters are litigated and still open? | 5 | correct · 15.5s | correct · 16.3s | correct · 10.2s |
| How many open claims have total payments exceeding their reserve? | 6 | correct · 26.3s | correct · 12s | correct · 11.3s |
| Which adjuster team has the highest total paid on litigated claims? | “Desk” | correct · 15.5s | correct · 13.2s | correct · 11.7s |
In-house legal department
Privileged material raises a waiver question as well as a policy one, which makes the boundary decision unusually consequential.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many documents are past their retention date but still under an unreleased legal hold? | 45 | correct · 21.4s | correct · 15.5s | correct · 10.4s |
| How many privileged documents belong to matters that are closed and have no unreleased hold? | 23 | correct · 21.9s | correct · 19.8s | correct · 11s |
| Which practice area has the most privileged documents? | “Commercial” | correct · 12s | correct · 11.8s | correct · 7.3s |
Three where it can
Most organisations are here, and many do not realise it. The data carries no residency constraint, so an approved hosted model is appropriate — faster to stand up, nothing to operate, and more accurate. We will say so rather than sell you infrastructure you do not need.
Online retailer
Order and inventory data carries no residency constraint, so an approved hosted model is the fastest and most accurate route.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many products are below their reorder point with no open purchase order? | 6 | correct · 12.8s | correct · 16.8s | wrong · 9.7s |
| Which category generated the most revenue from discounted orders? | “Outdoor” | correct · 16.5s | correct · 13.2s | correct · 11.5s |
| Which supplier has the highest proportion of sold items that were returned? | “Baltic Works” | correct · 18.1s | correct · 24.4s | correct · 16.6s |
B2B SaaS provider
Usage telemetry is commercially sensitive but not residency-restricted. Hosted is appropriate and quicker to stand up.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| Among accounts whose subscription is not cancelled, how many used fewer than half their licensed seats in 2026-07? | 27 | correct · 18s | correct · 12.7s | correct · 16.9s |
| How many active accounts had fewer active seats in 2026-07 than in 2026-05 and also have an unresolved support ticket? | 8 | correct · 18.7s | wrong · 16.2s | correct · 17.4s |
| Which plan has the highest total monthly recurring revenue among active subscriptions? | “enterprise” | correct · 12.3s | correct · 12.8s | correct · 13.2s |
County assessor’s office
Parcel, assessment and sale records are public. There is no residency argument to make, which makes this the clearest case that AI is not off the table.
| Question | Answer | OpenAI | DeepSeek | Local |
|---|---|---|---|---|
| How many parcels have a 2026 assessed value more than 25 percent below their most recent arm's-length sale price? | 6 | correct · 14.8s | correct · 22.8s | wrong · 17.2s |
| How many parcels with a pending appeal also had their assessed value rise from 2025 to 2026? | 4 | correct · 20.1s | correct · 17.7s | correct · 10.3s |
| Which neighbourhood has the most pending 2026 appeals? | “Cedar Heights” | correct · 16.3s | correct · 15.3s | correct · 9.8s |
Which model is your decision, not ours
Most organisations do not need a private model, and we will say so. If your policy already permits an approved AI vendor, the fastest and most accurate route is a hosted frontier model — no GPUs to buy, nothing to operate, and the accuracy shown above. That is our default, and for most clients it is the right answer.
What we will not do is pretend that choice is free of consequence. In that configuration your prompts, and the query results the agent reads, are sent to OpenAI under your agreement with them. For a great many workloads that is entirely acceptable and already covered by an existing vendor review. For some it is not, and that is the case we are unusual in being able to handle.
- Hosted — our default
- Most accurate, nothing to operate, priced per token. Data reaches the model provider. Right for most clients most of the time.
- On your hardware
- For data that genuinely cannot leave. Nothing reaches a third party and there is no per-token cost once the hardware exists. Accuracy depends heavily on which open model you run and on what hardware — the local column above is a single 30B model on one machine, and a larger model or better hardware would likely close much of the gap.
- Mixed
- Usually the right answer once a deployment matures: restricted data local, everything else through an approved API, decided per use case rather than per vendor.
- Switching later
- One line of configuration. You are not choosing once and living with it, and you are not locked to us either way.
One result is worth dwelling on, because it contradicts the obvious assumption. The hosted model was more accurate but slower here — 377s against 268s — because it ran with high reasoning effort. “Hosted is fast, local is private” is too simple. The real trade runs between accuracy, latency, cost and where your data goes, and it shifts with configuration.
Results vary by model, by hardware and by run. These figures are one measurement of two configurations on one date, not a guarantee. We have seen the same local model answer the same question correctly in one run and incorrectly in the next. If a specific model matters to you, the honest answer is to run these graded questions against it rather than trust a table on a website — and we will do that with you, on your schema.
Finish
Want this run against your schema?
An Assess engagement starts by establishing what your data actually permits — which is often less restrictive than people assume. Bring a schema and a question you cannot currently answer.