Going deeper

Score the use case before building

Five questions eliminate most bad projects:

• Variability: does the path really change from case to case? If not, write a workflow.
• Cost of error: what does a mistake cost, and can it be undone?
• Volume: is there enough repetition to repay the build?
• Data: are the tools and sources reachable and reliable?
• Measurable outcome: can you tell, without opinion, whether a task was done correctly?

The real cost per task

Tokens are often the smallest line. The full sum is the model cost, plus the human time for each escalation, plus the expected cost of errors:

cost = model + escalation rate × human cost + error rate × error cost

At $5 of human time per task, an agent that costs $0.10 per run, escalates 20% of cases and gets 1% wrong at $50 each comes to $1.60. Raise the error rate to 5% and the cost of each error to $500, and it costs $26.10. Same model, opposite decision. Accuracy is what drives the numbers.

Roll out in stages

1. Shadow: the agent runs on real requests, and its output is compared with what humans did. Nobody sees it, and it takes no action.
2. Assist: it drafts, and a person reviews each result. Track how often they change it: that’s your real error rate.
3. Canary: it acts for a small share of requests, with thresholds that trigger a rollback.
4. Autonomy, within limits: wider scope, with sampling reviews and hard caps.

if (killSwitch.on || budget.exceeded(run)) return handToHuman(request);
if (!canary.includes(request.userId)) return legacyFlow(request);
return agent.run(request);

Plan the off switch

A kill switch has to work without a deployment, and someone must be allowed to use it. Decide in advance what happens to runs in progress, and where requests go when the agent is off. If nobody can do the work by hand any more, you have no fallback, so keep that skill alive.

Watch for silent decline

Quality drifts when models, data or user habits change, even if your code hasn’t. Track the escalation rate, the human correction rate, the cost per task and a sample of reviewed results. Pin model versions, and rerun your evaluations before moving to a new one.