Putting Guardrails Around Autonomous AI Coding Agents
November 15, 2026 · 8 min read
An AI coding agent can make meaningful progress on a software task with very little supervision. It can read a ticket, inspect a repository, plan a change, edit files, run tests, and open a pull request. The same autonomy that makes this useful also creates a problem: an agent can keep working long after it has stopped making useful progress.
A coding agent might repeatedly run a failing test, rewrite the same function, ask a model for more context, or delegate work to another agent. Every action can consume tokens, time, and money. A system prompt that says "be efficient" is not a reliable control.
I would treat resource governance as a core part of the harness architecture. The agent proposes work; an external orchestrator decides whether that work is allowed to continue.
The agent should not control its own limits
The first design decision is to keep execution limits outside the agent.
An agent can be instructed to stop after a certain number of iterations, but the instruction is still part of the model context. The model can misunderstand it, ignore it, or lose track of it during a long run. The orchestrator should own the authoritative counters and enforce the limits before allowing another model call or tool action.
I would define budgets at several levels:
- Task budget: The maximum resources available to complete a Jira issue or other unit of work.
- Agent budget: The portion allocated to a planner, coder, tester, or reviewer.
- Model-call budget: The maximum input and output tokens allowed for an individual request.
- Tool budget: The allowed number or types of operations, such as shell commands, Git actions, or browser calls.
- Time budget: A deadline for the task and, where appropriate, for each agent execution.
- Cost budget: A monetary ceiling based on model pricing and infrastructure usage.
The values should be configurable by task type. A small documentation change should not receive the same budget as a multi-service feature.
Allocate a budget before work begins
Consider a task with a total budget of 50,000 tokens. The orchestrator might initially reserve capacity like this:
Task: Implement customer authentication
Total token budget: 50,000
Planning agent: 5,000
Architecture agent: 5,000
Coding agent: 25,000
Testing agent: 8,000
Review agent: 7,000
------
Total: 50,000
These numbers are illustrative, not recommended defaults. The important part is that the child allocations cannot silently exceed the parent task's budget.
I would allow the orchestrator to move unused capacity between stages when there is a good reason. For example, if planning finishes well under budget, some of its unused allocation could be made available to implementation. The overall task ceiling would remain unchanged.
For multi-agent systems, this needs to be hierarchical. A parent agent that delegates work to three child agents must reserve or allocate their budgets before those agents start. Otherwise, each child can appear to be within its own limit while the combined task exceeds the intended spend.
Track actual usage, not estimates alone
The budget manager should record usage for every model request:
- Input tokens and output tokens.
- Model identifier and provider.
- Request duration and outcome.
- Estimated or provider-reported cost.
- Task ID, agent ID, and parent execution ID.
- Retry number and reason for the retry.
Use provider-reported token usage whenever it is available. Local models may expose different usage details, so the adapter for each model runtime should normalize what it can and mark estimates clearly.
The budget check belongs in the orchestration path. Before another model request is made, the orchestrator checks the remaining allocation. If the request would exceed the limit, it can reject the request, reduce the allowed output size, switch to a cheaper model when appropriate, or stop the task.
A simplified control flow looks like this:
async def run_agent_step(task, agent, request):
budget = await budget_store.get(task.id, agent.id)
if budget.deadline_passed():
raise TaskTimeout(task.id)
if budget.iterations_remaining <= 0:
raise IterationLimitReached(task.id)
if budget.remaining_tokens <= 0:
raise TokenBudgetExceeded(task.id)
response = await model_client.generate(
request,
max_output_tokens=budget.max_output_tokens,
)
usage = normalize_usage(response)
await budget_store.record_usage(task.id, agent.id, usage)
if await budget_store.task_limit_reached(task.id):
raise TaskBudgetExceeded(task.id)
return response
This is intentionally simplified. A production implementation also needs atomic budget reservations, handling for failed requests, provider-specific usage fields, and a clear policy for whether a request that has started can finish after a budget threshold is crossed.
Use checkpoints instead of an endless agent loop
I would not give a coding agent an unrestricted loop that runs until it decides the work is finished. Instead, break the workflow into bounded steps and return control to the orchestrator between them.
A typical sequence might be:
- Read the ticket and acceptance criteria.
- Inspect the relevant repository and identify affected components.
- Produce an implementation plan.
- Make a bounded code change.
- Run targeted tests and static checks.
- Review the results and decide what to do next.
After each checkpoint, the orchestrator evaluates the state. It can allow another implementation cycle, request a plan revision, send the task to a reviewer, ask for human input, or stop.
This also makes the workflow easier to recover. If a worker crashes, the system can resume from a recorded checkpoint rather than restarting the entire task and paying for the same reasoning again.
Detect lack of progress
Token limits protect the budget, but they do not tell you whether an agent is stuck. I would add progress signals to detect repeated or unproductive behavior.
Examples include:
- The same tool call is repeated with the same arguments.
- The same test failure appears across several iterations.
- The Git diff remains unchanged after multiple coding steps.
- The agent repeatedly edits and reverts the same lines.
- The agent keeps requesting context that has already been provided.
- The task's acceptance criteria are not getting closer to completion.
No single signal proves that an agent is stuck. A test can legitimately fail several times while a complex issue is being debugged. The goal is to combine signals and trigger a checkpoint, not automatically terminate every task that repeats an operation.
A useful policy is to distinguish between a hard limit and a warning threshold. A hard limit stops execution. A warning threshold asks the orchestrator to reassess the plan, summarize progress, or request approval for additional resources.
Manage context as well as tokens
Token consumption is not only about how many times an agent calls a model. Long conversations can become expensive because the same history is repeatedly included in later requests.
I would keep durable task state outside the conversation:
- The original requirements and acceptance criteria.
- The current implementation plan.
- Decisions and constraints.
- Files inspected and files changed.
- Test results and unresolved failures.
- A concise summary of the current state.
At a checkpoint, the agent can produce a structured handoff summary. The next execution can load that summary and the specific files or test output it needs, rather than replaying every prior message.
This is especially important when handing work from a planner to a coder or from a coder to a reviewer. Each agent should receive the context needed for its responsibility, not the entire transcript by default.
Give the system a safe stopping path
When a budget is exhausted, the system should not simply disappear or leave a half-finished task in an ambiguous state.
It should record:
- What was completed.
- What remains incomplete.
- The last known-good commit or checkpoint.
- The reason execution stopped.
- Token and cost usage.
- Any tests that passed or failed.
- What approval or additional information is needed to continue.
For Git-based workflows, I would keep changes isolated on a task branch and avoid merging automatically just because the agent ran out of budget. The system can preserve the work for review while keeping the protected branch unchanged.
A practical architecture
The main components can remain fairly small:
Task API / Jira
|
Orchestrator
|
Resource Governance
- Token budgets
- Cost ceilings
- Timeouts
- Iteration limits
- Tool permissions
|
Agent Workers
/ | \
Planner Coder Reviewer
\ | /
Checkpoints
|
Build / Test / Git
|
Execution Records
The orchestrator owns state transitions and budgets. Agents perform bounded work. Tool adapters enforce permissions. The execution record provides the information needed to resume, debug, and audit a task.
Takeaways
- Do not rely on a prompt to control an agent's spending. Enforce limits in the orchestrator.
- Use hierarchical budgets so delegated agents cannot collectively exceed the parent task's allocation.
- Track actual token usage, cost, time, iterations, and tool calls for every execution.
- Return control to the orchestrator at explicit checkpoints.
- Detect lack of progress and repeated failures, but use multiple signals rather than one simplistic rule.
- Keep task state and summaries outside the conversation so agents do not repeatedly consume tokens on old context.
- Make stopping safe: preserve the work, report what happened, and require approval before exceeding the budget.
Autonomy does not mean giving an agent unlimited authority. The more independently an agent can act, the more important it becomes to put clear limits around what it can spend, change, and execute.