Essay

Making AI actually useful in production

February 2025

The demo always works. You assemble the right context, write the right prompt, stream the output to a terminal, and it looks like magic. That's where most AI projects live — and die.

Getting an LLM to do something impressive once is a solved problem. Getting it to do something useful reliably, at 3am, when your team is asleep and the stakes are real — that's a different problem entirely. I learned this building Transformers, the incident-response agent swarm. The model was never the hard part.

The gap is everything around the model

The gap isn't model capability — the model is usually good enough. It's everything else: context that arrives incomplete, a downstream tool that times out, a confident answer that's wrong, an invocation bill ten times your spreadsheet.

The demo-production gap: a small demo box on the left, and a production boundary on the right containing the same model box surrounded by evals, tracing, fallbacks, a cost ceiling, and on-call. The demo prompt → output PRODUCTION Evals Tracing prompt → output Fallbacks Cost ceiling On-call
The demo-production gap: in production, the model is the smallest box in the system.

Production isn't a stress test of the happy path. It's a relentless exploration of the failure modes you didn't design for. A bug in a function is deterministic; a hallucination isn't — so you build for observation instead.

What "useful" actually means

Before any AI feature, ask one question: what does useful mean here, specifically? Not impressive. Not clever. Useful — to a specific person, in a specific moment, in a way they'd notice if it broke.

For Transformers, that person is the on-call engineer. Three dimensions matter:

  • Reliability. The bar depends on stakes: a writing assistant that misfires costs a shrug. An incident summary that misfires at 3am costs trust you won't earn back.
  • Observability. You can see what the system did, why, and where it went wrong. Without this, debugging AI is archaeology.
  • Cost discipline. The unit economics work at the scale you actually run. Fine at 100 invocations a day can be catastrophic at 10,000.
Dimension The bar for the on-call engineer What breaks without it
Reliability An incident summary you'd trust at 3am. One wrong summary and trust is gone for good.
Observability See what it did, why, and where it went wrong. Debugging the agent becomes archaeology.
Cost discipline Unit economics that hold at real scale. Fine at 100/day, catastrophic at 10,000.
"Useful" for one person, on three axes

Patterns that held up

Pattern What fails without it How it shows up in Transformers
Narrow, verifiable scope Agents that do too much can't be tested. One agent owns one slice — logs, metrics, deploys — and only the orchestrator reasons across.
Tracing from day one Debugging AI is archaeology. Trace every call from week one; a wrong summary is pinpointed in minutes.
Graceful failure over false confidence Silent failure stitches a plausible summary. Explicit fallback at every tool call; on-call gets "I couldn't get the log data", not a hallucination.
The three patterns that held up

Narrow, verifiable scope

The worst AI features try to do too much; the best do one verifiable thing. Each Transformers agent owns one slice — logs, metrics, recent deploys, related services — and only the orchestrator reasons across the whole picture. So you can test each agent in isolation, and fix it when it breaks.

The Transformers swarm: a central orchestrator fans out to four narrow agents — logs, metrics, recent deploys, and related services — each with its own fallback exit, and emits one summary. Tracing runs as a bus beneath every call. Orchestrator reasons across logs metrics recent deploys related services ↘ fallback ↘ fallback ↘ fallback ↘ fallback incident summary tracing · every call on the bus
One orchestrator, four narrow agents — each with a fallback exit, all on a tracing bus

Tracing from day one

We traced every call from the first week, as the way we understood the system — not an afterthought when something broke. Every agent call, every tool invocation, every intermediate result, logged and queryable. When a summary came out wrong, the trace showed exactly which agent produced which output — failure pinpointed in minutes. Without traces, you're guessing.

Graceful failure over false confidence

The most dangerous AI system is one that fails silently. If a tool call times out, you want the agent to say "I couldn't get the log data for this service" — not stitch a plausible summary from partial data. We built explicit fallbacks at every tool call.

The same partial-data input — logs missing — taken two ways. On the left, false confidence: the agent stitches a plausible but wrong summary, the danger. On the right, graceful failure: the agent reports "I couldn't get the log data" honestly, the behavior we built. partial data in log tool timed out FALSE CONFIDENCE "Root cause: a memory leak in checkout." plausible · stitched · wrong → trust gone GRACEFUL FAILURE "I couldn't get the log data for this service." honest · bounded · on-call can act
Same missing logs, two outputs: a confident hallucination vs. the honest fallback we built

The lesson I keep relearning

AI in production isn't a machine learning problem. It's a systems design problem. The model is a component — a powerful, weird, probabilistic one — but still just a component. It needs what every critical component needs: defined failure modes, observability, graceful degradation, and a clear answer to "what does success look like and how would we know?"

Get that right, and the model will surprise you with what it can do. Get it wrong, and you'll spend your nights putting out fires the model lit.