"Add AI to the product" is now on almost every roadmap. The hard part isn't calling a model API; that takes an afternoon. The hard part is choosing the right approach for your problem, then making it reliable, affordable and safe enough to put in front of customers.
Here's how I walk business owners through that decision.
Three approaches, in order of cost and complexity
1. Prompting a hosted model. You send instructions and the user's input to a model like Claude or GPT and use the response. This covers a surprising amount: drafting emails, summarizing documents, classifying support tickets, extracting fields from messy text. Always start here.
2. Retrieval-augmented generation (RAG). When the model needs to know your information, such as product docs, policies, past tickets or a client's files, you store that content in a searchable index (often Postgres with pgvector), retrieve the relevant pieces for each question, and include them in the prompt. The model answers from your data instead of guessing.
3. Fine-tuning. You train a model further on your own examples so it adopts a specific format, tone or narrow skill. It's useful in specific cases, but it's the most expensive to build and maintain, and it does not reliably teach a model new facts. That's what RAG is for.
How to choose
- The task is general and the model already knows how to do it → prompting.
- Answers must come from your own documents or data, and that data changes → RAG.
- You need a very consistent output style at high volume, and prompting plus examples isn't enough → consider fine-tuning, after you've proven the feature with prompting.
- You need the AI to take actions, like booking, refunding or updating records → tool use (function calling), with strict limits on what each tool can do.
Most products that succeed with AI end up with good prompts, a solid RAG pipeline and well-scoped tools. Very few genuinely need fine-tuning on day one.
What makes an AI feature production-ready
A demo that works on five hand-picked examples is not a feature. Before launch, an engineer should have:
- An evaluation set. Dozens to hundreds of real examples with known good answers, run automatically whenever prompts or models change, so you can tell if a change made things better or worse.
- Grounding and citations. For RAG, the answer links back to the source passages, so users and your support team can check it.
- Guardrails. Input validation, output checks, and refusal paths for questions the system shouldn't answer.
- Least-privilege tools. An AI agent that can issue refunds should have a refund limit, an audit log and, above a threshold, a human approval step. It should never hold production database credentials.
- Privacy controls. Know which data goes to which provider, strip what isn't needed, and check the provider's data-retention terms against your obligations.
- Cost controls. Cache repeated requests, pick the smallest model that passes your evals, set per-user rate limits, and alert on spend.
- Latency handled in the UI. Stream responses, and move long-running AI work into background jobs instead of blocking the request.
- Observability. Log prompts, retrieved context, outputs and user feedback (with sensitive fields redacted), so you can debug a bad answer after the fact.
Common mistakes I see
- Starting with fine-tuning because it sounds more serious.
- Stuffing an entire knowledge base into one giant prompt instead of retrieving what's relevant.
- Shipping without evals, so every prompt tweak is a guess.
- Letting the model generate SQL or call APIs with unrestricted permissions.
- Treating the model's confidence as accuracy. Models can be fluent and wrong.
The bottom line
Start with the simplest approach that could work, measure it against real examples, and add complexity only when the numbers tell you to. The model is the easy part. The engineering around it, meaning retrieval, evaluation, guardrails and cost control, is what turns a demo into a feature your customers can trust.
- AI AGENTS
