Frame, orchestrate, evaluate, ship: how I take AI from idea to production

“Agent” has become one of the most stretched words in technology. It gets attached to chatbots, scripts with a model call inside, and genuinely autonomous systems alike. That makes it hard to decide where agents belong in an organisation, and easy to build one where something simpler would do better.

Here is the definition I work with: an agent is a language model, a set of tools and a loop. The model looks at the goal and what it knows so far, decides on the next step, calls a tool, reads the result and repeats until the job is done or it hands over to a person. Everything else (memory, planning, multiple agents talking to each other) builds on that loop.

Workflow or agent?

The first question is not which agent to build, but whether you need one at all.

If you can draw the steps in advance, build a workflow: a fixed sequence where a model handles specific steps, such as reading a document or classifying a request, and ordinary code handles the rest. Workflows are predictable, cheap to run and easy to test.

Reach for an agent when the path depends on what you find along the way. Investigating an incident, chasing down why two records don’t match or assembling context from several systems are hard to script, because the next step depends on the last result.

In practice, most production systems I build are workflows with agentic pockets: a deterministic backbone, with an agent handling the steps that genuinely need judgement.

Where agents earn their place

The strongest use cases I see share three traits: high volume, messy inputs and a clear definition of a good outcome.

• Triage and classification. Reports arrive in different formats and languages, and each one needs a category, a priority and an owner. An agent can read, enrich and route them in seconds, so people spend their time on the cases that matter.
• Investigation. Pulling context from case histories, documents and databases to prepare a decision is exactly the kind of multi-step, path-dependent work agents handle well.
• Reconciliation. Matching records across systems and explaining the exceptions turns a periodic manual exercise into something that can run continuously.
• Document review. Flagging documents that look manipulated, incomplete or inconsistent, for a person to check, rather than making the final call.

Notice that none of these remove the human. They change where the human spends their attention.

The parts that matter more than the model

When an agent misbehaves, the model is rarely the root cause. These usually are:

1. Tools. Each tool should do one thing, be clearly described and only have the permissions it needs. Standards such as the Model Context Protocol (MCP) make it much easier to expose systems to agents consistently, but the design discipline is still yours.
2. Context. An agent can only reason about what it can see. Decide what it retrieves, what it remembers between runs and what it should never see.
3. Guardrails. Spell out what the agent may never do, cap how many steps and how much budget it can use, and require approval before anything irreversible.
4. Observability. Log every step and tool call. When something goes wrong, you need to replay the run, not guess.
5. Evaluation. A set of real cases with known good outcomes, run before every change. Without it, you are tuning by anecdote.

Humans in the loop, on purpose

“Human in the loop” is often added at the end as a safety blanket. It works better when it is designed in from the start, at the points where judgement or accountability actually sits:

• approval before actions that are hard to undo, such as payments, external messages or record changes
• a review queue for low-confidence results, with the agent’s reasoning attached
• a clear escalation path when the agent is stuck, instead of letting it try forever

The goal is not to remove people. It is to move them to the decisions that need them.

The same holds when the agent writes the code

AI coding agents now write much of the code on my client builds. The pattern is the same as in any agentic system: the agent handles volume, and I own the framing, the architecture, the review and the final engineering call. The agent is fast; the direction still has to come from someone who understands the problem and is accountable for the result.

A short checklist before you build one

• Can you write down what a good outcome looks like, and check it on real examples?
• Could a workflow do the job? If so, start there.
• Does each tool have the narrowest permissions that still work?
• Where are the human checkpoints, and who staffs them?
• Can you replay any run from its logs?
• Who owns the system after launch?

If you can answer all six, an agent is very likely worth building. If you can’t, the gaps are where the project would have stalled anyway.

A new language model is released every few weeks, each one topping some leaderboard. It is tempting to treat model choice as the main decision in an AI project and to switch every time the rankings move.

In my experience, the leaderboard rarely decides what works in production. Your tasks, your data and your constraints do. The model is one component of the system, and the system is what has to deliver.

Think in tiers, not names

The major providers offer families of models in roughly three tiers, and they are worth thinking about separately:

• Frontier models are the most capable, and the slowest and most expensive. Use them for complex reasoning, long multi-step agentic tasks and difficult code.
• Mid-tier models handle the bulk of production work well: drafting, summarising, extraction with judgement, most tool use.
• Small, fast models are ideal for classification, routing, simple extraction and anything you run at very high volume.

Anthropic’s Claude models, for example, come as Opus, Sonnet and Haiku; OpenAI and Google offer similar ranges. Alongside them sit open-weight families such as Llama, Mistral and Qwen, which you can host yourself.

Good systems often use several tiers at once: a small model triages and routes, and a larger one handles the cases that need real reasoning. Matching the model to the step usually saves more money than any discount negotiation.

Evaluate on your own work

Public benchmarks measure general ability. They tell you little about how a model handles your documents, your terminology and your edge cases.

What works is a small evaluation set built from real examples:

1. Collect 50 to 200 real cases, including the awkward ones.
2. Write down what a correct result looks like for each.
3. Score each model on accuracy, the kinds of mistakes it makes, whether it follows the required format and how reliably it uses tools.
4. Run the same set every time a new model comes out, or whenever you change a prompt.

An afternoon of evaluation beats a week of reading release notes.

Once that set exists, deciding whether to adopt a new model takes hours instead of weeks of debate.

The criteria that actually decide

When I compare models for a production system, these are the questions that matter:

• Quality on your evaluation set. Not in general; on your tasks.
• Tool use and instruction following. For agentic work, the model must call tools correctly, stay within its scope and stop when it should. This varies more between models than raw intelligence does.
• Latency. A model that is a little better but twice as slow can make an interactive product feel broken.
• Cost per completed task. Count the whole task, not the price per token: long prompts, retries and multiple steps add up. Prompt caching and batch processing can change the picture significantly.
• Context handling. A large context window is not the same as using that context well. Test retrieval from long inputs explicitly.
• Data protection. Where is data processed, how long is it retained, is it used for training, and is there a proper data processing agreement? For European organisations, this often narrows the choice before quality is even discussed.
• Reliability. Rate limits, regional availability and how the provider handles outages.

Hosted or open-weight?

Hosted frontier models give you the most capability with the least operational effort. Open-weight models give you control: data never leaves your infrastructure, you can fine-tune, and costs are predictable at scale.

The trade-off is operational effort and, for the hardest tasks, usually some capability. A common and sensible split is hosted models for complex reasoning and open-weight models for high-volume or particularly sensitive steps.

Design for change

The one thing I can predict is that today’s best choice will not be next year’s. So I build systems where switching models is a configuration change followed by an evaluation run, not a rewrite:

• a thin layer between the application and the model provider
• prompts and evaluation sets kept under version control, next to the code
• the model for each step set in configuration, not hard-coded

With that in place, a new release becomes an opportunity rather than a migration project.

Where I start

My default is to start each step with a strong mid-tier model, measure it on real cases, then move the hardest steps up a tier and the high-volume, simple steps down. I re-run the evaluations whenever a significant new model arrives, and only switch when the numbers say so.

The model matters. But the evaluation set, the tool design and the guardrails around it are what turn a good model into a system people can rely on.