A demo only has to work once. A feature in front of customers has to work every time, including on the bad days. Before an AI feature goes live, I check five things.
- What happens when the model is slow or down. Every model and API call has a timeout, a retry and a fallback, and the user sees something sensible instead of a spinner.
- What happens when it is unsure. Low-confidence answers go to a person, and the hand-over is part of the design, not an afterthought.
- What it is allowed to touch. The agent only has the tools and data it needs, and every action that changes money, records or customer data is logged.
- What it costs per request. Cost is tracked per call from day one, so the first monthly bill is not a surprise.
- How we will know it broke. Logs, traces and alerts are in place before launch, so a failure reaches a person before it reaches a customer.
None of these make a demo look better. All of them decide whether the feature is still running a month after launch.