Strategy

From pilot to production: a field guide for shipping AI that lasts

6 min read By Lexior Group

The demo works. The room is impressed. Then the pilot ends, the team disperses, and the workflow returns to the way it was.

That gap is not a failure of the model. It is a failure to treat production as a different problem. A pilot proves that a model can produce a plausible answer. Production proves that an organization can rely on that answer on a busy Tuesday, at volume, with an audit trail, and with someone accountable when it is wrong.

The failure modes are predictable. Teams that plan for them ship systems that last. Teams that ignore them spend a year rebuilding the same prototype.

Why pilots stall

The reasons are usually organizational as much as technical, and they tend to appear in a familiar order.

No clear owner

A pilot can survive on borrowed enthusiasm. A production system cannot. If no business leader owns the workflow, no one can decide what “good enough” means, approve a change to the process, or defend the budget when the novelty fades. The pilot team ships a prototype and waits for an owner who was never named.

No evaluation

Pilots are often judged by demonstrations. A room watches a handful of prompts, the answers look right, and the project is declared a success. Without a fixed set of test cases and agreed acceptance criteria, no one can say whether the next model, prompt or retrieval change makes the system better or worse. Teams optimize by anecdote, and quality drifts.

Data access arrives late

The pilot reads a spreadsheet exported for the occasion. Production needs live data, the same permissions as the people using it, a refresh schedule and a retention policy. Each requirement touches systems the pilot never had to integrate with. Discovering this after the demo turns one project into two.

Security review happens at the end

When the architecture is fixed before review, the review becomes a negotiation. Data flows, vendor terms, logging and access control are easier to shape at the start than to retrofit. Bring security and privacy into the first design session, not the last gate.

No operating model

A demo has no on-call rotation. Production does. Someone must watch quality, cost, latency and failures; update prompts and models; respond to incidents; and decide when to retire a feature. If that work has no owner and no allocated time, the system decays from the day it launches.

DimensionPilotProduction
QuestionCan the model do this?Can the organization rely on this?
SuccessA convincing demonstrationMeasured quality on agreed cases
DataA one-time exportLive access with permissions and retention
SecurityReviewed at the endDesigned in from the start
ChangePrompts edited by handVersioned, tested and reviewed
OperationsNo owner after handoffNamed owner, monitoring and on-call
CostEstimated or ignoredTracked per workflow and reviewed

Choose the first use case for production, not for the demo

The best demo is often the worst first production system. A general assistant that answers anything is impressive on stage and difficult to evaluate, secure and improve. A narrower workflow is less exciting and far more likely to ship.

Use these tests to choose the first candidate:

First-use-case checklist

  • Frequency. The workflow happens often enough that improvements matter.
  • Bounded scope. Inputs and outputs are clear enough to test.
  • Tolerable failure. Mistakes can be caught by a person or a rule before they cause harm.
  • Accessible data. The documents and systems already exist in a usable form.
  • A measurable baseline. You can describe how the work is done today and what better means.
  • A named owner. A business leader will use the result and fund the next phase.

If a candidate fails two or more of these tests, it is a research project, not a first release.

Define success and evaluation before you build

Evaluation is the discipline that turns an opinion into evidence. It starts with a set of real cases, drawn from the workflow with the consent of the people who own them. For each case, the team records what a good answer looks like, which sources are acceptable, and which failures are unacceptable.

A practical evaluation set

Start with a few dozen cases and grow the set as the system meets real traffic. A useful set covers routine work, edge cases and known traps. It includes the questions users actually ask, not only the ones that are easy to answer.

For each case, write down:

  • the expected answer, or the criteria for a good one;
  • the sources that count as authoritative;
  • the behavior that is unacceptable, such as inventing a citation or ignoring a permission;
  • the person who can arbitrate when reviewers disagree.

Review the set before the first release and again after every serious failure. Each failure becomes a new test case, and each new capability gets cases that would catch its characteristic mistakes. This is unglamorous work, and it is the difference between a system that improves and one that merely changes.

Measure more than accuracy

Quality is not one number. Track accuracy, grounding in sources, latency, cost per completed task and the rate at which people accept the output. A system that is slightly less accurate but fast enough to use may be the better product. A system that is accurate but slow enough to be abandoned is not a product at all.

Design for production from day one

Production architecture is mostly about what happens around the model. The model is one dependency; the system around it is what makes the dependency safe to use.

  • Retrieval that respects permissions, so an answer never cites a document the user cannot open.
  • A model gateway that centralizes keys, quotas, retries, fallbacks and logging.
  • Versioned prompts and configurations, reviewed like code.
  • Separate environments for development, testing and production.
  • Traces that record the question, the sources retrieved, the model and the response.
  • Cost controls: budgets, rate limits and alerts before the invoice arrives.
  • A fallback path when the model is unavailable or confidence is low.
  • Data retention rules that match the promise made to users.

None of this is exotic. It is the same discipline we apply to any system that touches customer data. The mistake is to treat AI as an experiment that will somehow acquire these properties later.

Treat the model as a replaceable part

Models improve, prices change and vendors retire versions. A production system should let you swap the model behind a stable interface without rewriting the application. Keep prompts, retrieval and evaluation outside vendor-specific code, and test each candidate model against the same cases. The team that can change models cheaply is the team that keeps getting better.

Plan operations and ownership

Launch is a beginning. The operating model should name who does what before the first user arrives.

  • A product owner accountable for outcomes and priorities.
  • An engineering owner accountable for reliability and change.
  • A review process for prompt, model and retrieval changes.
  • A monitoring routine with thresholds for quality, latency, cost and errors.
  • An incident path for harmful, wrong or unavailable outputs.
  • A quarterly decision to continue, expand or retire.

Write this down. A one-page operating agreement is worth more than an architecture diagram no one maintains.

Iterate with real users

Release to a small group, in shadow mode where possible, and compare the system’s output with what people actually do. Watch where they ignore it, correct it or work around it. Those moments are the real specification.

Resist the temptation to add features before the core workflow is trusted. A system that does one job reliably earns the right to do more. A system that does many jobs adequately earns nothing.

Set a cadence: weekly review of failures and feedback, monthly review of cost and usage, quarterly review of the use-case portfolio. Retire what does not earn its place.

Summary

Key takeaways

  • A pilot proves a model can answer; production proves an organization can rely on the answer.
  • Choose a narrow, frequent workflow with a named owner and a measurable baseline.
  • Build the evaluation set before the product, and run it before every release.
  • Treat permissions, logging, cost and fallbacks as design requirements, not launch chores.
  • Name the operating model — owner, review cadence, incident path — before the first user arrives.

If your team has a pilot that will not move, the problem is usually not the model. We help leaders choose the right first use case, design the evaluation and operating model, and build the production system around it. Start with AI strategy or generative AI applications; when the platform itself needs work, our cloud and MLOps practice takes it from there. Tell us where you are today and we will give you a candid view of what is worth building.

Start a conversation

Let’s turn AI ambition into systems you can rely on.

Tell us where you are today. We reply within one business day with a candid view of what is worth building — and what is not.

Or call +1 (714) 900-0052