Evaluation first: taking AI features from demo to production
Most AI prototypes fail in production for the same reason: nobody can measure quality. Here is how we build the measurement before the feature.
Photo: Negative Space / Pexels
Almost every AI project we are asked to rescue has the same history. A prototype impressed in a meeting, a few weeks went into polishing prompts, and then real users arrived with real data. Quality dropped, nobody could say by how much, and every fix seemed to break something else.
The missing piece is rarely the model. It is measurement.
Start with fifty real examples
Before we write a prompt, we collect real inputs from the client’s own data: support tickets, documents, form submissions. Subject-matter experts label what a good answer looks like. Fifty well-chosen examples are enough to start, and the set grows to several hundred as edge cases appear.
Make every change answer to the set
Prompt edits, retrieval tweaks and model swaps are all experiments. Each one runs against the evaluation set and reports a quality score, a cost and a latency. If a change does not beat the current version, it does not ship.
- Accuracy against expert labels
- Cost per request at expected volume
- Latency at the 95th percentile
- Failure modes, grouped and counted
Design for the cases the model gets wrong
No model is right every time, so the product has to handle being wrong gracefully: citations users can check, confidence that is visible, and a human approval step wherever an action is hard to undo.
A demo answers “can it work?” An evaluation set answers “does it work, how often, and at what cost?”
That second question is the one your customers, your finance team and your regulators will eventually ask. It is far cheaper to answer it from the first week than to reconstruct the answer after launch.
Working on something like this? We would like to hear about it.
Start a Project
