I switched off my father's phone line one morning and let an AI answer it instead.
It lasted a few hours. In that time it took orders for things the kitchen does not make, quoted prices that do not exist, and agreed to combinations the store does not allow. Every call sounded excellent. The AI was warm, it was fast, it never got flustered, and a customer listening to it would have said it was the best phone experience the restaurant had ever offered.
Then the tickets came out of the printer and none of them were real.
What it actually cost
The money was the smallest part. Comped orders and remakes for one morning is a number you can absorb.
What I had actually spent was credibility, and I spent three kinds of it at once. My father's store lost credibility with regulars who ordered something, were told yes, and then were told no by a human twenty minutes later. I lost credibility with my father, who had trusted me with the front of his business. And in a very small way, AI lost credibility with a few dozen people in Baltimore who now believe this technology does not work.
That third one is the one nobody budgets for. Every bad automated interaction spends down a shared account, and it is the reason “it demoed well” is not an acceptable standard.
The mistake was structural, not a prompting problem
Here is what I had built, and what almost everybody builds first. Speech to text, a language model in the middle, text to speech on the way out, and a database write at the end. Give the model the menu in its prompt. Tell it to be accurate. Ship it.
The reasoning seems sound. The model has the menu. Why would it invent an item? Because it is not looking anything up. It is producing the most plausible next thing to say, and a plausible answer to “can I get that half mushroom, half onion, uncut, with jalapeños on the whole thing” is yes. It is a friendly, helpful, entirely reasonable yes. It is also wrong, if the store does not split toppings by half or send pizzas out uncut.
And people order in wildly different shapes. “Hey! Can I get a large pizza? Can I get it half mushroom, half onion? And can you add jalapeños on the whole thing?” versus “Hello. Can I get a large jalapeño pizza with half onion and half mushroom, but don't cut it.” That is the same pizza. A human handles both without thinking. A language model also handles both, which is exactly the problem, because it will handle the second one into an order that cannot be made.
I tried to fix this the way everyone tries first. I wrote a better prompt. I listed the rules. I told it what it could not do, in bold, twice. It got better, which is worse than useless, because “better” means the failures move somewhere you are not looking. You cannot instruct your way out of this. You are asking the model to be reliable about the one thing it is structurally unreliable about.
The fix is the opposite of what the field is doing
Everything about the last three years has pushed toward giving models more freedom, more tools, more autonomy. For anything that touches money, I think that is backwards.
The way to make an AI system credible is to constrain it.
Not to make it dumber, and not to script the conversation. The conversation should stay completely free, because that is the half the model is genuinely extraordinary at. The model owns understanding a caller who changes their mind mid-sentence, handling interruptions, recognising that “actually make that two” refers to the thing mentioned eleven seconds ago. Deterministic code owns what exists, what it costs, what is legal, and whether anything actually happened.
In my system the model cannot name a menu item by writing its name. It proposes typed edits against opaque references, and application code resolves those references against the real catalog, checks whether the resulting operation is legal, prices it, and writes it. The model can be as creative as it likes and still cannot invent a product, a price, or a successful order, because it has no vocabulary in which to express one.
A rule the model is told to follow is a suggestion. A rule it cannot express is a guarantee.
What this costs, honestly
It fails closed. If a caller asks for something the catalog cannot describe, the system does not improvise. It says it cannot do that, or it routes to a human. This means my agent sometimes refuses things a person behind the counter would have said yes to. In a demo that looks like a limitation. In production it is the entire point, because a refusal is recoverable in ten seconds and a silently wrong order is not recoverable at all.
A concrete case I like, because it is small. Somebody says “no mayo.” That is not a priced modifier and it is not on the menu. The lazy options are to drop it silently or to treat it as a topping and charge for it. Both are wrong in ways that surface later at the counter. So it stays attached to the line as an unpriced note and routes the order to manager review. The system does not pretend to know what it does not know.
A few other boundaries fell out of the same principle. Any material change to an order invalidates whatever confirmation came before it. Partial speech can trigger a lookup but can never authorise a write, because reads are free to be wrong and writes are not. An interruption stops the agent talking, but it does not by itself cancel a business action, because those are two different events and treating them as one loses orders.
The general version
If you are putting a model in front of anything transactional, the useful question is not “how do I stop it hallucinating.” It is “which categories of truth must never be model-owned.” My list, after a year of this: price, availability, policy, payment, acceptance, and success. The model may discuss all six. It may assert none of them.
Everything else can stay free, and it should. The tone, the phrasing, the recovery when somebody changes their mind, the ability to sound like a person rather than a phone tree. That is the half worth having, and constraining the other half is what makes it safe to ship.
What I am still not certain about
Every failure mode in this system is one I thought of. That is the limit of it. The failures that will matter are the ones a few hundred real callers find in the first month, and by definition I do not know what those are yet. Bounded proof calls are not a Friday dinner rush.
And the claim I would push hardest on if I were reading this is the catalog one. Configuration living in data rather than code is what makes a second restaurant a data-entry job instead of a rewrite. I believe that. It is proven on one menu. A store with a genuinely different structure is the real test and I have not run it.
The system goes live in September. The thing I am most confident about is not the accuracy number. It is that when it does not know something, it will say so.