Never Let the AI Touch the Data

The first version of every AI data assistant works like magic and lies to your face.

You paste in a dataset - say, income by town across Israel - wire it up to a capable model, and ask: "Which town has the highest average salary?"

Back comes a beautiful answer: a town name, a number, a little explanation.

And sometimes it is completely wrong. Not wrong like a typo - wrong like the town isn't in the top ten and the number appears nowhere in your file. The model didn't look anything up. It pattern-matched a plausible answer and delivered it with the same confidence it uses for the correct ones. That's the whole problem: a language model asked to be a calculator will guess, and you can't tell its guesses from its facts.

I ran into this while building the AI chat on top of the datasets in Simpler Story, where people wanted to ask questions about a dataset in plain Hebrew instead of reading a chart. The problem showed up immediately: if the AI makes up even one number, you can no longer trust the ones it gets right, and a tool meant to help people read data becomes a way to mislead them.

The fix isn't a better prompt. It's an architecture that never lets the model near the arithmetic. To see why that's the only thing that reliably works, it helps to be precise about what LLMs are actually bad at.

LLMs are not databases

It's tempting to stuff the rows into the prompt and let the model answer. For a tiny table it even seems to work. But "read this data and compute the answer" is the one thing a language model is structurally bad at:

• It doesn't reliably count. Ask it to sum a column of 400 numbers and it produces a number - one that looks right and usually isn't. There's no adder inside, just a next-token predictor doing an impression of one.
• It silently loses rows. Anything past the first screenful gets skimmed, averaged-by-vibes, or dropped, and you have no idea which.
• It confuses units and definitions. Gross vs. net, monthly vs. annual, median vs. mean - the model mixes them, because in its training text those words all hang out together.
• It doesn't naturally stop at "I don't know." Left to narrate freely, it fills gaps with fluent invention.

Better prompts and bigger models help around the edges, but they don't fix the underlying problem: the part of the system writing prose shouldn't also be the part doing the arithmetic. So instead of trying to make the AI better at the data, I kept it away from the data entirely.

Try this yourself before you trust it with a spreadsheet

Don't take my word for it - it takes thirty seconds to watch it happen. Open your favorite chat model and ask it to multiply two numbers that are ugly enough not to be memorized:

How much is 5.37779 * 8.284884?

You'll get a clean, confident answer, complete with a "more precisely" tail of decimals that makes it look like real computation happened.

Now type the same multiplication into an actual calculator:

The model didn't run out of precision; it never computed anything. Expand its reasoning and you can watch it decide arithmetic is beneath it: "This is a straightforward multiplication problem. I can calculate this directly without needing any tools." Then it hallucinates a plausible product and dresses it up with fake extra decimals.

If it can't be trusted with one multiplication of two numbers you can see, ask yourself what it's doing to a column of 400 numbers you can't. That's the whole reason for the rule that follows.

The AI never computes from the rows. It translates the question into a query, deterministic code runs that query, and the AI only narrates a result it wasn't allowed to invent.

How it works: the AI as translator, not analyst

Move the AI to the edges and put boring, verifiable code in the middle:

question
  → AI translates
structured query
  → deterministic code executes
computed result
  → AI narrates
grounded answer

In Simpler Story the assistant never sees the full raw table. It sees the dataset's schema: column names, field types, labels, units, and a few example values. From that it produces a constrained query object - which column to rank by, which filter to apply, which aggregation to run - and the app executes it. The AI shows up in exactly two places, neither of them "doing math on the data":

1. At the front, as a translator, turning "which town pays best?" into that query object. It's using what it's genuinely great at - mapping fuzzy language onto a precise schema.
2. At the back, as a narrator, handed a small computed result (the town, the value, the units) and asked only to phrase it. It can't get the number wrong, because it isn't producing the number.

The concrete stack, for the curious: Opus orchestrates that translate-execute-narrate loop, and Gemini handles the more visual, design-leaning generation. The specific models barely matter, though. You can swap in whatever's best this quarter and nothing changes, because the model is kept away from the arithmetic either way. None of the reliability depends on the model being good at math.

Three boundaries I had to get right

1. Helpful vs. honest

A chatbot desperately wants to be helpful - and that instinct is exactly what makes it dangerous over data, because the most helpful-feeling move (answer everything, always) is how it lies. So I deliberately made it less accommodating. If a question maps to no valid query, or the data doesn't contain the answer, the right response is "I can't answer that from this dataset" - full stop, no improvising.

That feels like a downgrade the first time you watch it refuse, but it isn't. An assistant you have to double-check is worse than the plain table it replaced. Refusing to answer when it can't verify something is what makes the rest of its answers usable.

2. Open-ended questions vs. constrained execution

People ask anything; the execution layer can only rank, filter, aggregate, and compare over known columns. The gap is where the work lives. The AI translates toward that menu and is only allowed to emit a query naming real columns and known operations - never arbitrary code against my data. "Which town pays best" is a clean rank. "Why does the periphery earn less" is not a query but an interpretation - so the system offers context, not an invented causal number. The small executable surface is what makes it safe to point at real data.

3. Natural narration vs. verified numbers

Even handed a correct result, a chatty model loves to add - a comparison you didn't compute, a "roughly double" that's actually 1.4×. Every embellishment is a fresh unverified number. So the narrator is kept on a short leash: phrase the computed result, add only context that's true by construction, and introduce no new figures. This was the hardest of the three to get right, because a well-behaved narrator still has to sound natural. It just never reaches for a number nobody computed.

There's a useful payoff to confining the AI to phrasing: different readers want different registers. A researcher wants it academic and hedged; a newsroom wants it punchy. Since the narrator only ever rewords a number it can't change, I can let it shift tone freely on request and still show the same verified figure underneath every version.

If you're building one

The pattern that worked for me: put deterministic code on anything that has to be correct, and let the AI own only the fuzzy edges, understanding the question and phrasing the answer. The design starts from what the model isn't allowed to do rather than what it can do; the whole architecture here fell out of "it never touches the rows." And make "I can't answer that" a real, supported response, because an assistant that can't say it will invent something rather than sit in silence.

None of this is clever, and that's the point. The impressive version, where you paste in the data and let the model answer, is the one that eventually hands someone a confident wrong number. I'd rather ship the boring one: a model wired to a calculator it can't override, doing only the parts it's actually good at.