A log saying that an AI agent restarted a failed service proves very little. It does not prove that the intended service changed, that no neighbouring workload moved, that capacity recovered, or that the agent still held valid authority when it acted.
That evidence gap becomes expensive once agents can drain GPU nodes, change scheduler policy, alter storage mounts or roll out a new model-serving image. A human operator can interrogate an ambiguous dashboard before proceeding. An unattended agent may interpret missing data as a healthy system and continue.
A new paper from Jun He and Deying Yu proposes agent-native telemetry: typed, signed evidence built around state changes rather than prose log lines. The work is an early preprint, and GPUMachines has not reproduced its benchmarks. Still, it asks the right purchasing question. Before an organisation lets an agent operate live AI infrastructure, can the platform prove what changed, why it changed and whether the evidence stream was complete?
For most buyers, the answer today is no. The remedy is not another dashboard. It is an operations design that separates agent intent from execution authority and records the resulting infrastructure delta outside the agent's control.
The purchasing answer
Do not give an AI agent unattended write access to production GPU infrastructure merely because it can read OpenTelemetry traces, call Kubernetes tools and produce a convincing incident summary. Those capabilities support diagnosis; they do not create trustworthy change control.
Production autonomy should wait until each state-changing run can bind together:
- the requesting human, service or upstream agent;
- the exact intent and scope approved for the run;
- the infrastructure state observed before execution;
- the tool call or control-plane action that took effect;
- the measured state delta and post-change health;
- any gap, dropped record or uninstrumented boundary;
- the rollback decision and its outcome.
The evidence recorder should not depend solely on the same agent it audits. High-impact changes should use scoped, short-lived authority, policy checks and a rollback path that has been tested on the real platform.
Teams building a private operator can start with the GPUMachines Private Agent Fleet solution. The infrastructure brief should include the evidence plane, management network and change-control path alongside model serving. If the project will manage several compute nodes, use the GPU Cluster Configurator to establish the physical cluster boundary before deciding where the autonomous control services will run.
What the paper proposes
The paper, submitted to arXiv on 17 August 2026, describes an Agent Telemetry Protocol (ATP) and a State-Delta Evidence Ledger. Its basic unit is not a free-form log message. It is one of four typed records:
| Evidence primitive | What it records | GPU infrastructure example | |---|---|---| | Transition | A discrete change in state or execution outcome | Compute node moves from allocated to drained | | Observation | A measurement, probe result or invariant check at a point in time | Fabric error counter crosses an agreed limit | | Relation | A link that was added, changed or removed | Model service now depends on a different cache endpoint | | State checkpoint | A bounded snapshot with sequence and drop information | Active nodes, jobs, power limits and collector coverage at a known time |
Schemas are content-addressed. Change a field, unit or permitted value and the schema receives a different digest, which helps expose silent semantic drift. Producers assign ordered sequence numbers, group records into batches, construct a Merkle root and sign the batch. Each batch also points to the previous root.
The collector verifies the signature, schema digest, sequence and chain continuity before appending a batch. A separate checkpoint channel retains the accepted chain head, allowing a reader to detect when ledger storage loses or hides a suffix. At read time, a verifier labels the requested range as complete, truncated, tampered or gapped.
That last distinction matters. An agent should only conclude that an event did not happen when the instrumentation profile covered the relevant producers, schemas and time range without an unresolved gap. The paper calls this ledger-relative verified non-occurrence. It cannot prove what uninstrumented code never reported, and the authors say so.
The design exposes verified data to agents by two paths. A stateless decoder emits compact positional rows. A stateful gateway maintains a versioned entity graph and returns a bounded evidence capsule around the service, dependency or incident under investigation. Free-form stack traces and request strings live behind digest-checked references so an agent does not ingest attacker-controlled prose unless it asks for that detail.
Reported results, with the caveat they need
The authors evaluated a prototype on AIOpsLab HotelReservation and the OpenTelemetry Astronomy Shop. Their test environment used a 16-node Kubernetes cluster and 120 injected incidents across crash loops, memory leaks, latency, packet loss, configuration drift and deployment regressions.
Relative to structured OpenTelemetry JSON, the paper reports:
| Author-reported result | ATP configuration or test | |---|---| | 96.4% lower wire payload and modelled query-scan cost | State-delta representation | | 88.8% fewer LLM context tokens | Stateful semantic gateway | | 66.2% fewer agent query operations | Stateful semantic gateway | | All 500 adversarial storage mutations detected | Tampering tests | | No successful prompt injections in 50 trials per ATP configuration | Opaque-text isolation tests |
The evaluation also reports higher diagnostic F1 scores and shorter root-cause identification times for the gateway configuration. These are the authors' measurements, not independent benchmarks. They used model versions including Claude 3.5 Sonnet, GPT-4o, Llama 3.1 and Qwen 2.5, so a buyer should not assume the same token use, accuracy or latency with a newer model and a different tool loop.
There is another restraint: the publication describes roughly 16,000 lines of implementation across producer, collector, verifier and access-layer components, but the arXiv record does not provide an obvious public code repository. Until code, test fixtures and operational instructions are available for outside inspection, ATP should be treated as a research architecture rather than a ready procurement standard.
The figures are interesting because they quantify a real problem. They cannot support replacing an established telemetry stack during a procurement cycle.
Why ordinary observability is not enough
Conventional observability answers useful questions: which request was slow, which pod restarted, how much GPU memory a process consumed, or where a trace crossed a service boundary. It was designed mainly for people and deterministic software.
An infrastructure agent adds two different questions:
1. Did the evidence presented to the agent cover the system and period on which its conclusion depends? 2. Can an independent reviewer connect the approved intent to the action that took effect and the state that followed?
A JSON log can be structured yet still fail both tests. It may omit records during a network partition, change meaning after a schema update, contain an attacker-controlled error string, or sit in storage that an administrator can edit. A trace can show that the agent called a tool without proving that the target accepted the change. An agent's own final summary is weaker still because it is a model-generated interpretation of the run.
AWS's Agentic AI Lens recommends structured traces for reasoning, tool calls and hand-offs, plus immutable and privacy-safe audit trails. An IETF draft on auditing agent delegation goes further by separating interaction, action, delegation and authorisation-transition records. Its authors argue that an agent which records itself can provide useful telemetry but cannot alone provide non-repudiation to an outside auditor.
Those sources approach the problem from different directions, yet they agree on the operational fault line: observing an agent is not the same as proving the effect of an action.
What this means for a GPU cluster
GPU estates give an autonomous operator many ways to create an expensive partial failure. The obvious risk is powering off the wrong node. More likely problems are subtler: draining a node that holds the only warm model replica, changing a scheduler label that strands jobs, rolling a driver to an incompatible subset of hosts, moving storage traffic onto a congested path, or applying a power cap that reduces throughput without tripping a health alarm.
Each change needs domain evidence.
Node drain and job movement
Before a drain, capture the node identity, active jobs, local model or checkpoint state, topology role, requested maintenance window and evacuation capacity. The state delta should identify every workload that moved or terminated. Post-change evidence should confirm that replacement placements meet the intended GPU, CPU, memory and fabric constraints.
"Drain succeeded" is not enough when a distributed job has lost one rank and continues to hold seven accelerators idle.
Scheduler and quota policy
A change to Slurm, Kubernetes or a higher-level queue can alter who receives scarce GPUs. Evidence should retain the old policy digest, approved new digest, authorising identity, affected queues or namespaces and a replayable record of the transition. Post-change checks should inspect pending jobs, placement failures, pre-emption and tenant isolation rather than merely confirming that the API returned HTTP 200.
Fabric and network changes
Routing, VLAN, RoCE, InfiniBand or link-policy changes can divide a cluster without taking every service offline. The relevant state includes topology, port role, peer identity, error counters, congestion signals and the management path used to execute the change. A separate IETF network-operations draft states that exchanged state must not itself imply authority to act; configuration, filtering, route and rollback changes still need the deployment's authorisation and audit process.
An agent should never convert "I can see the network" into "I may rewrite it".
Storage and checkpoint paths
Autonomous remediation may remount a filesystem, fail over metadata services, change a client route or evict cached data. The evidence contract should cover mount options, namespace identity, client reachability, active writers, capacity, metadata health and an application-level read or write probe where appropriate. A green storage daemon does not prove that training jobs can still reach their checkpoints.
Model-serving rollouts
A serving agent may change container image, model revision, quantisation, tensor-parallel layout, routing weight or cache policy. Record each as a typed change, then measure health against the service objective: loaded replicas, model identity, output checks, time to first token, queue depth and error rate. Our analysis of agentic AI serving and KV-cache scheduling explains why a software routing change can make one GPU hot while another remains idle.
The infrastructure behind trustworthy autonomy
Agent-native telemetry does not remove ordinary infrastructure requirements. It adds an evidence plane that must survive the incident it is meant to explain.
Separate control, evidence and workload paths
The model-serving network, storage fabric, management network and evidence path should have defined roles. They do not always need physically separate switches, but the design must prevent a saturated training or storage flow from silently cutting off the collector that certifies coverage. Out-of-band management remains valuable because an agent may need to inspect or recover a node whose operating system path has failed.
The signing service, collector and independent checkpoint destination should not share one failure domain by accident. Placing them all in the same Kubernetes cluster makes deployment easy, but a control-plane outage can then remove the operator and its evidence together.
Identity and time are part of telemetry
Every producer needs a stable identity, protected signing material and a way to rotate keys. The system also needs bounded clock uncertainty. A timestamp alone cannot establish order across several nodes when clocks drift; sequence continuity and boot epochs carry part of that burden.
Agent identity should be distinct from the human or service on whose behalf it acts. Sub-agent delegation should narrow authority rather than copying one broad bearer token through the tool chain. The audit record then needs to retain which authority was in force at the moment of execution.
Size hot state separately from retained evidence
The semantic gateway needs enough CPU and RAM to maintain the active graph, evaluate invariants and answer bounded queries. The append-only evidence tier optimises for ordered writes, integrity checks and retention. Large opaque objects such as stack traces, heap profiles and packet captures suit a separate object tier linked by digest.
These workloads have different retention and access patterns. Keeping all of them in an expensive searchable log service can waste money; collapsing all of them into cold object storage can make incident response too slow. Buyers should model event rate, batch size, checkpoint frequency, hot investigation window, retention obligation and expected agent query volume.
The paper's 184 MB gateway memory figure came from one microservice testbed with 12,400 active state vectors and 45 invariant rules. It is not a sizing ratio for a GPU cluster. A cluster inventory may have fewer entities but richer per-node topology, job, storage and fabric relationships.
Keep the reasoning model outside the evidence root
The LLM may summarise, rank hypotheses and choose its next query. It should not define the canonical history after the fact. Model-generated root-cause notes belong in a derived plane that points back to exact evidence ranges and records the model, prompt, tool version and policy used.
This separation lets a team replace the model without rewriting what happened. It also allows a deterministic verifier or human engineer to challenge the agent's conclusion against the same signed records.
Security boundaries the paper does not remove
Hash chains detect some storage mutations; they do not make a compromised producer truthful. If malware controls the application before an event reaches the producer SDK, it can omit or falsify what the SDK signs. Stolen signing keys weaken provenance. A rogue collector and untrusted checkpoint service may collude. The paper places compromised producers, collectors and application-level omission outside its cryptographic guarantee.
Typed schemas also need governance. An authorised but badly designed schema can omit the field that matters. A legitimate update may redefine an enum or unit. Content addressing exposes the change, but someone still has to review and approve the new schema and decide how old and new records compare.
Privacy creates another tension. Immutable evidence can retain user identifiers, prompts, paths or operational secrets long after an incident. Redaction and pseudonymisation must happen before signing because later editing would break the record. The opaque store needs access controls and deletion policy even when the ledger keeps its digest.
Prompt-injection isolation is sensible, not absolute. Once the agent deliberately fetches an untrusted stack trace or request body, delimiters and trust labels can reduce risk but cannot prove that a model will ignore hostile instructions. A high-impact action still needs policy enforcement outside the model.
A staged deployment path
Start with observation, not remediation. Give the agent read-only access to a bounded evidence gateway and compare its diagnoses with the on-call team's incident record. Track unsupported claims, missed dependencies and queries made against incomplete coverage.
Next, allow the agent to draft a proposed state delta and rollback plan without execution rights. A deterministic policy layer should check target scope, maintenance window, current health, protected workloads and whether the evidence range is complete. A human approves the proposal.
Low-risk, reversible actions can move to supervised execution after the team has measured success and failure on the real estate. Examples might include collecting a diagnostic bundle or restarting a stateless test service; the acceptable set will vary by organisation. Power, network, storage, identity and fleet-wide software changes should remain behind stronger approval until failure tests show that authority, evidence and rollback survive partial outages.
For owned hardware in a managed facility, GPUMachines Buy & Host can provide a deployment route where power, cooling and remote hands are defined while the buyer retains the server asset. That does not outsource agent governance. It does give the project a clearer physical and operational boundary than an improvised lab cluster.
What to put in the procurement acceptance test
Ask vendors and internal platform teams to demonstrate the evidence path, not merely show an agent completing a happy-path task.
1. Identity: Trace one run from the approving principal through the agent, sub-agent and tool to the target service. 2. Authority: Show the exact scope and expiry of the credential used for the change. Attempt an out-of-scope action and confirm that the control plane rejects it. 3. Before state: Retrieve the signed pre-change checkpoint and the dependencies relevant to the proposed action. 4. Actual delta: Compare the intended change with the state transition observed at the target boundary. 5. Coverage failure: Drop a producer or collector path. The agent must report a gap rather than infer health from silence. 6. Tampering: Mutate, reorder and truncate retained records in a test environment, then verify that reads fail with an explicit status. 7. Opaque input: Place hostile instructions in a diagnostic string. Confirm that canonical processing isolates it and that fetching it cannot bypass action policy. 8. Rollback: Induce a failed post-change invariant and execute the agreed reversal while preserving both attempts in the audit trail. 9. Recovery: Restart the agent, gateway and one collector component. Confirm that boot epochs, sequence continuity and outstanding acknowledgements recover correctly. 10. Human reconstruction: Give an engineer the retained evidence without the agent's summary. They should be able to establish intent, authority, action and outcome.
If the platform cannot pass the gap test, it is not ready for unattended action. A polished explanation cannot compensate for missing evidence.
Who should consider this architecture
The strongest candidates operate private GPU clusters, multi-tenant inference services, research estates or edge fleets where agents will eventually touch schedulers, model services, networks or storage. Regulated or sovereign environments may also value an audit record that remains under their control and can be checked without trusting the model vendor.
It is equally relevant to a smaller team planning ahead. Designing stable entity IDs, management separation and signed change records before the fleet grows costs less than retrofitting them after several agents and tools have adopted incompatible logs.
Who should not build it yet
Do not create a cryptographic telemetry platform to automate a single workstation that one engineer can inspect directly. Standard system logs, configuration management and human approval may be enough.
Teams with poor inventory, no tested backups, shared administrator credentials or an undocumented network should fix those controls first. An evidence ledger cannot repair an unknown physical topology or a rollback that has never worked.
And do not replace OpenTelemetry simply because a preprint reports smaller payloads. Existing metrics, logs and traces still serve people, vendor support and established incident tools. A transition-centred evidence layer may sit beside them, consume selected signals or start with one bounded workflow. It needs to earn wider adoption.
Frequently asked questions
Is agent-native telemetry the same as AI agent observability?
No. Agent observability usually records model calls, prompts, tool use, cost, latency and internal workflow. Agent-native infrastructure telemetry records what the external system did and whether the evidence is complete. A production design needs both, joined by stable identity and trace context.
Does ATP replace OpenTelemetry?
The paper compares ATP with OpenTelemetry representations, but production buyers should treat it as an architectural proposal. OpenTelemetry remains useful for human operations and software tracing. A state-delta ledger could coexist with it, especially while teams validate instrumentation coverage and tool support.
Can signed logs prove that an agent made the right decision?
They can prove that certain records came from identified producers and remained in sequence under the stated trust model. They cannot prove that the policy was wise, that an uninstrumented event did not occur, or that a compromised producer told the truth. Decision quality still needs evaluation and human accountability.
Does an agent need a GPU to analyse telemetry?
Not always. Structured evidence can reduce context and may work with smaller local models or deterministic rules for bounded tasks. The model choice depends on query complexity, latency, privacy and concurrency. Keep inference capacity planning separate from the collector and evidence-store requirements.
Where should the evidence ledger run?
Place it where it can remain available and verifiable during the failures being investigated. That may mean dedicated management nodes, a separate cluster, or a hosted control service with an independent checkpoint destination. Avoid putting every component inside the same failure domain as the workloads under control.
Should an autonomous agent be allowed to roll back its own change?
Only under an explicit policy with bounded scope and a tested rollback. The evidence system should preserve the failed change, trigger, approval state, rollback action and result. For network, storage, fleet-wide software or power changes, human confirmation may remain the sound choice.
Can GPUMachines design the hardware around a private operations agent?
GPUMachines can review private inference capacity, management-node roles, network separation, storage, rack power, hosting and cluster topology. The buyer still needs to define the agent's permissions, evidence policy and acceptance tests with its operations and security teams.
Sources and Further Reading
- Agent-Native Telemetry paper by Jun He and Deying Yu
- IETF draft: An Architecture for Auditing AI Agent Delegation and Interactions
- IETF draft: Operational Requirements for Network State Exchange in Agent-Assisted Network Operations
- AWS Agentic AI Lens: Observability and monitoring for agentic systems
Verdict
Agent-native telemetry identifies the right boundary for autonomous infrastructure: the operator must reason over evidence that is typed, scoped and independently checkable, not simply consume more prose logs. The new ATP paper gives that idea a concrete protocol, ledger and evaluation, but it remains an early research result without an obvious public implementation for outside validation.
Buyers should take the architecture seriously and the benchmark numbers cautiously. Build a small evidence-backed control loop around one reversible workflow, force it through coverage loss and rollback, then decide whether wider autonomy deserves access to the production estate.
For a private deployment, review the GPUMachines Private Agent Fleet solution and map the agent, management services, evidence store, network boundaries and GPU workloads before selecting the final hardware.