Adding AI to an App Without the Hype

An AI feature is easy to demo and hard to ship. The demo hides latency, cost and failure — the three things that decide whether the feature survives contact with real users. This is how we build AI features that keep working.

Design the feature as a service, not a scattered call

The most common mistake is calling the model directly from UI code. Instead, wrap all model access behind a single service layer. That one seam owns the prompt, the retries, the caching, the timeouts and the error surface. It gives you one place to add streaming, one place to add a fallback, and it makes the behaviour testable without a UI.

class AiService {
  async complete(input) {
    try {
      return await withTimeout(model.call(input), 8000);
    } catch (e) {
      return fallback(input); // never block the user
    }
  }
}

Build the fallback first

Models go down, rate limits hit, networks drop. If the app is unusable when the model is unavailable, you built a demo. We build and ship the non-AI path first, then treat AI as the enhancement. Users forgive a good fallback; they don't forgive a frozen spinner.

Control cost with hybrid routing

Not every task needs a large model. We route cheap, repetitive tasks (classification, validation, simple intent) to a small on-device or cheap cloud model, and call the large model only when the task genuinely needs it. This cuts cost dramatically and makes most interactions instant. We also cache aggressively — identical prompts should not cost twice.

Measure quality like an engineer

"It looked good when I tried it" is not evaluation. We build a small golden set of real inputs with known-good outputs — including the ugly cases: ambiguous inputs, no-answer questions and adversarial prompts. Then we track accuracy, hallucination rate and latency on every change. RAG systems get their retrieval recall and answer faithfulness measured separately, because when an answer is wrong you need to know instantly whether it's retrieval or generation.

Guardrails for anything that leaves the device

If the AI output goes to a customer, a document or a database, it needs guardrails: schema validation, refusal when confidence is low, and moderation on generative output. And on the security side, never let untrusted content trigger tool calls directly — prompt injection is a real attack. Run tools with least privilege and confirm anything destructive.

Watch it in production

Before launch we instrument the AI path specifically: success rate, latency distribution, fallback frequency and cost per active user. Without these numbers, tuning is guessing. With them, it's engineering. When latency spikes or cost creeps, you see it before your users do.

The honest summary

Good AI features are mostly ordinary software engineering: a clean service seam, a reliable fallback, cost control, honest measurement and guardrails. The model is the easy part. Everything around it is what makes the feature production-grade — and it's exactly the work that separates a demo from a product.