How to Build an AI Product That Works in Production

Pick the smallest technique that solves the problem

Most business AI use cases need retrieval over your own data plus a carefully designed prompt. Fine-tuning is worth it when you need consistent structure or tone at volume; retrieval wins whenever answers must reflect private, current, changing information.

Start with prompting, add retrieval when the model needs facts it cannot know, and consider fine-tuning only when you have data showing the first two are not enough.

Evaluation turns opinions into engineering

Without an evaluation set, every prompt change is a guess. Build a fixed set of real inputs with expected outcomes, score each release against it, and you can change models or prompts with confidence instead of superstition.

• →A golden set of 50–200 real cases with graded expected answers.
• →Automatic scoring where possible, human review for the rest.
• →Regression checks in the release process, not after user complaints.
• →Logging of inputs, outputs and user corrections, with consent and retention limits.

Cost, latency and failure behaviour

Unlike normal features, AI has a per-use cost. Model your gross margin before launch: tokens per action, actions per active user, users per plan. Then design it down with caching, smaller models for easy cases, and strict output limits.

Decide what happens when the model is slow, wrong or unavailable. A graceful fallback and a visible way for the user to correct the system is what makes AI feel trustworthy rather than flaky.