Measuring AI Coding Agents: From Token Usage to Engineering Quality
November 15, 2026 · 11 min read
Once an AI coding harness has more than one agent, debugging it becomes a distributed-systems problem. A Jira issue may move through a planner, a coding agent, a test agent, and a reviewer. Each agent can call a model, invoke tools, modify files, and hand work to another agent.
If a task takes twenty minutes or produces a poor pull request, I want to know where the time went, what it cost, which agent made which decisions, and whether the delivered code actually met the requirements.
Counting tokens is a start, but it is not a measure of engineering quality. I would build observability and evaluation into the harness from the beginning, with a clear separation between operational telemetry and code-quality measurement.
Two different questions to answer
I think about agent observability in two layers.
Operational observability answers questions such as:
- How many model requests did the task make?
- How many input and output tokens did it consume?
- Which tools were called, and how often?
- How long did each agent run?
- Where did retries and failures occur?
- What did the task cost?
Engineering evaluation answers different questions:
- Did the implementation satisfy the acceptance criteria?
- Did the tests pass?
- Did the change introduce regressions?
- Did static analysis or security scanning find problems?
- How much rework was needed?
- Was the pull request accepted, and how much human correction was required?
The first layer tells me whether the system is operating efficiently. The second tells me whether it is producing useful software.
Instrument the entire task with distributed traces
I would use OpenTelemetry as the instrumentation foundation. It provides a common way to represent traces, metrics, and logs across services and tools.
The key is to create a trace for the overall development task and connect the work performed by each agent to that trace.
For example:
Trace: JIRA-1042
|
+-- Planning agent
| +-- Read Jira issue
| +-- Inspect repository
| +-- Generate implementation plan
|
+-- Coding agent
| +-- LLM request
| +-- Read authentication files
| +-- Edit auth service
| +-- Run unit tests
|
+-- Testing agent
| +-- Run integration tests
| +-- Report one failure
|
+-- Coding agent (retry)
| +-- Fix token validation
| +-- Run targeted tests
|
+-- Review agent
+-- Inspect Git diff
+-- Run static analysis
+-- Produce review findings
Each operation should have a trace or span with useful attributes: task ID, agent role, model, operation name, repository, branch, and outcome. I would also record the parent-child relationship between agent executions.
That makes it possible to open one task and see the full chain of work instead of searching through unrelated logs.
I would be careful not to put secrets, full prompts, source code, or sensitive customer data into telemetry by default. Store only what is needed for diagnosis, apply access controls, and define retention policies. If prompts or tool payloads are captured for debugging, they need explicit redaction and data-handling rules.
Track model and tool usage
For each model request, capture as much of the following as the provider or runtime exposes:
| Measurement | Why it matters |
|---|---|
| Input tokens | Shows context consumption |
| Output tokens | Shows generation volume |
| Model and provider | Enables model comparisons |
| Request duration | Shows latency |
| Estimated or reported cost | Enables cost analysis |
| Retry count | Reveals unstable calls |
| Completion status | Shows request failures |
| Task and agent IDs | Connects usage to the work |
Tool calls deserve the same attention. A coding agent might spend more time running shell commands, reading files, or waiting for tests than it spends generating code.
For each tool call, I would capture the tool name, duration, result status, retry count, and a safe summary of the operation. This can reveal patterns such as an agent repeatedly running the full test suite when a targeted test would have been enough.
A normalized event might look like this:
{
"task_id": "JIRA-1042",
"agent_id": "coding-agent-01",
"agent_role": "coder",
"model": "provider-model-id",
"operation": "llm.generate",
"input_tokens": 8500,
"output_tokens": 3200,
"duration_ms": 42000,
"tool_calls": 12,
"retry_count": 1,
"status": "completed"
}
The model name and token fields are examples. In a real implementation, the adapter should preserve the provider's actual model identifier and distinguish reported usage from estimated usage.
Connect the telemetry to Jira and Git
An execution trace is much more useful when it connects to the engineering workflow.
I would carry a consistent task identifier through the system:
Jira issue
|
v
Orchestrator task ID
|
+--> Agent execution IDs
|
+--> Git branch
|
+--> Commit SHA
|
+--> Pull request ID
|
+--> CI build and test results
|
+--> Review and acceptance outcome
This correlation lets me answer questions that are otherwise difficult to investigate:
- Which agent produced this commit?
- How much did the implementation cost?
- Which model calls led to the change?
- How many test failures occurred before the pull request was ready?
- Was the pull request accepted without major rework?
I would not use a Jira ticket number as the only trace identifier. A task can have several execution attempts, and a pull request can contain multiple commits. Keep distinct IDs for the business task, the overall execution, each agent run, and the Git artifacts, then link them together.
Measure code quality independently
This is the part I consider most important: an agent should not be the only judge of its own work.
A model can report that it completed a task, but that does not prove the implementation is correct. I would build an evaluation pipeline that runs independently of the coding agent.
For a feature such as customer authentication, the pipeline could check:
- Acceptance criteria: Are the requested behaviors implemented?
- Unit tests: Do the relevant functions and components behave correctly?
- Integration tests: Do the services work together?
- Regression tests: Did existing behavior break?
- Static analysis: Are there maintainability, reliability, or code-smell issues?
- Security checks: Are there known vulnerabilities, unsafe patterns, or dependency risks?
- Human review: Does the implementation fit the architecture and make sense to maintain?
Tools such as SonarQube, language-specific linters, dependency scanners, and SAST tools can provide objective signals. Test results provide another independent signal. Human review remains important for design choices and requirements that are difficult to express as automated checks.
Build an evaluation scorecard
I would avoid reducing all agent quality to one magic number. Instead, expose several dimensions and make the evidence behind each score visible.
| Dimension | Example measurement |
|---|---|
| Functional correctness | Acceptance tests passed |
| Test quality | Relevant tests added and passing |
| Regression risk | Existing tests that now fail |
| Maintainability | Static analysis findings |
| Security | Scanner findings and severity |
| Efficiency | Tokens, cost, and elapsed time |
| Reliability | Retries, failures, and completion rate |
| Rework | Changes requested after initial implementation |
| Delivery | Pull request accepted and merged |
A dashboard could show task success rate, first-pass test rate, average cost per accepted task, average cycle time, and the number of tasks requiring substantial human intervention.
The definitions matter. For example, a first-pass test rate should specify whether it means all tests passed on the first run, or only the newly added tests passed. A task completion rate should define what counts as completed and whether abandoned or blocked tasks are included.
Without consistent definitions, comparisons between agents or models are misleading.
Compare models and agents using the same tasks
Once the harness records execution and quality data, it becomes possible to compare different models and agent strategies.
For example, I might compare two models on a set of representative repository tasks:
- Did they satisfy the same acceptance criteria?
- How many tests passed?
- How many defects were found during review?
- How many tokens and dollars did each task consume?
- How much time elapsed?
- How often did a human need to intervene?
The evaluation set should contain tasks of different sizes and difficulty. Comparing models on unrelated Jira tickets can produce misleading results because one model may have received much easier work.
For a more controlled comparison, use a fixed benchmark set, the same repository state, the same instructions, and the same test suite. Repeat runs when model behavior is variable. Keep the evaluation results tied to the model version, prompt version, tool configuration, and harness version.
This is also how I would evaluate changes to the harness itself. If I change the system prompt, add a reviewer agent, or alter the context-retrieval strategy, I want to know whether the change improves outcomes rather than just making the agent appear more active.
Choose an observability stack
For a Python-based harness, I would start with a relatively small set of components:
| Component | Role |
|---|---|
| OpenTelemetry | Instrumentation standard for traces, metrics, and logs |
| OpenTelemetry Collector | Receives and routes telemetry |
| Langfuse | LLM-focused tracing and evaluation workflows |
| Grafana | Dashboards and visualization |
| Tempo | Distributed trace storage |
| Loki | Log aggregation |
| Prometheus | Metrics collection and time-series queries |
I would not necessarily deploy every component on day one. The important thing is to standardize instrumentation and task correlation first. The storage and visualization backends can evolve as the system grows.
Langfuse can provide an LLM-oriented view of traces and model interactions. OpenTelemetry gives the broader instrumentation model that can span the API, orchestrator, workers, test runners, and other services. They solve related but not identical problems.
Be careful with the metrics you optimize
There are a few tempting measurements that can encourage the wrong behavior.
Lines of code generated is not a productivity metric. More code can mean more complexity and more maintenance.
Number of tool calls is not inherently good or bad. A complex task may need many calls, while an inefficient agent may make many unnecessary ones.
Test coverage is useful, but high coverage does not guarantee meaningful tests or correct behavior.
Task completion rate can be gamed if the system allows agents to mark tasks complete without independent validation.
I would focus on outcomes: accepted features, verified acceptance criteria, escaped defects, rework, cycle time, and cost. Operational measurements explain why those outcomes occurred.
Measure cost per accepted feature
One metric I particularly like is cost per accepted feature:
Cost per accepted feature =
Total model and infrastructure cost
------------------------------------
Features accepted
This is more meaningful than cost per model call or cost per line of code. It connects AI spending to an engineering outcome.
I would pair it with cycle time and defect escape rate. A system that produces inexpensive pull requests but creates a large review burden is not necessarily delivering value.
Takeaways
- Instrument the full workflow, not just model requests. Agent orchestration, tool calls, builds, tests, and Git operations all matter.
- Use trace IDs to connect Jira issues, agent runs, commits, pull requests, and CI results.
- Track token usage, latency, cost, retries, and tool activity as operational telemetry.
- Evaluate code independently with tests, static analysis, security scanning, and human review.
- Compare models using consistent tasks, repository states, prompts, and evaluation criteria.
- Avoid vanity metrics such as lines of code generated. Measure accepted work, quality, rework, cycle time, and cost.
- Keep sensitive prompts, source code, and tool payloads out of telemetry unless there is a deliberate, secure reason to capture them.
An AI coding platform needs to answer two separate questions: how efficiently are the agents operating, and how good is the software they deliver? Distributed tracing helps explain the first. Independent engineering evaluation helps answer the second. Both are necessary before I would trust an autonomous coding workflow with meaningful production work.