Do Agentic AI Clusters Need CPU Racks?

Agents spend time in tools, sandboxes, retrieval and orchestration as well as models. Trace the task before turning that CPU work into a separate rack.

Buying another GPU will not shorten a tool call that is waiting on Python, a database or an isolated sandbox. Agentic AI makes that obvious because the model is only one worker in a longer chain of orchestration, retrieval, code execution, policy checks and application logic.

That does not mean every agentic deployment needs a dedicated CPU rack. It means the CPU tier has to be measured as deliberately as the accelerator tier.

The buying question is: are CPU-side services consuming enough time, memory bandwidth or concurrency to delay the GPU service, and do they need to scale or fail independently? A small private system may keep these functions on the GPU hosts or a few general-purpose servers. A busy multi-agent platform can justify separate CPU racks when thousands of tool calls, sandboxes and workflow steps run beside dense model inference.

AMD and NVIDIA both used their 2026 platform material to argue that agentic workloads change the CPU/GPU balance. Their products take different approaches, but the procurement lesson is the same: count completed agent tasks, not GPUs in isolation.

GPUMachines has not independently benchmarked the newest EPYC 9006 or Vera CPU platforms. Vendor performance comparisons are not used here as sizing ratios. The specifications and architecture descriptions should be checked against the exact production system and software release.

The short answer by deployment size

One AI workstation or one GPU server: keep orchestration, retrieval and light tool execution on the same machine unless isolation, reliability or memory pressure says otherwise. A second modest CPU server is often a better first step than a dedicated rack.

A small research cluster: use the CPUs already present in GPU nodes for data preparation and host duties, then add a shared services tier for schedulers, databases, repositories and sandboxes. Separate racks are optional.

A production agent platform with sustained concurrency: separate CPU capacity becomes attractive when tool execution, retrieval, workflow engines and application services need independent scaling. This can be a few dense CPU nodes before it becomes a full rack.

A large AI factory: dedicated agentic CPU racks can be justified when CPU work is a measured part of task latency and power, and when operating teams need separate failure, security and deployment domains.

The phrase “CPU rack” should therefore describe a scaling outcome, not the starting architecture.

What runs outside the model

An agent request may appear to be one inference job, but its critical path can include:

1. authentication and policy evaluation; 2. prompt construction and context retrieval; 3. model prefill and token generation; 4. parsing a tool call; 5. launching a container, virtual machine or sandbox; 6. executing code or querying a service; 7. validating the result; 8. writing state to memory or a database; 9. calling the model again; 10. logging actions for audit and billing.

Only some of those stages require a GPU. Many prefer high single-thread performance, many cores, large system memory, fast storage or low-latency connections to data services.

Agentic systems also create uneven work. A model may produce a short tool call in milliseconds, then the tool runs for seconds. A research agent may fan out across dozens of web or document tasks. A coding agent may compile a project, run tests and start containers. A customer-service agent may wait on a transactional database. Average CPU utilisation can look low while a particular queue dictates user-visible latency.

This is why CPU sizing from “cores per GPU” alone is unreliable. The ratio changes with the number of agents, tool mix, sandbox lifetime, retrieval design, model serving architecture and service-level objective.

Four CPU roles that should not be mixed blindly

GPU host CPUs

Every PCIe GPU server and most rack-scale systems need CPUs to run the operating system, feed accelerators, manage I/O and host the model runtime. These processors are coupled to the GPUs by platform design. Undersizing them can restrict data preparation, networking or storage even if external CPU racks exist.

Orchestration CPUs

Schedulers, API gateways, workflow engines and request routers coordinate work. They often need predictable per-core latency rather than maximum core count. A slow orchestration path leaves GPUs idle between steps.

Sandbox and tool-execution CPUs

Code interpreters, browsers, compilers, simulations and isolated containers can consume many cores and large memory footprints. High concurrency often favours dense-core nodes, while individual interactive tools may care more about single-thread speed.

Data and application CPUs

Vector databases, document processing, feature extraction, graph search, transactional services and cache layers have their own memory, storage and latency patterns. Placing all of them in one generic “CPU pool” can turn one noisy service into a platform-wide problem.

Separate racks are useful only after these roles have been identified. Otherwise a buyer simply moves an unknown bottleneck into a larger enclosure.

What AMD and NVIDIA are proposing

AMD describes agentic infrastructure as a distributed system with GPU racks for model compute and CPU racks for orchestration, processing data and tool execution. Its EPYC 9006 range spans high-frequency and dense-core options. AMD lists up to 256 cores and 512 threads, up to 16 memory channels on selected platforms, and PCIe Gen6 connectivity. These are platform capabilities, not a recommendation to use the highest-core part for every role.

NVIDIA positions the Vera CPU around high loaded per-core performance, memory bandwidth and efficient execution of agentic services. Vera Rubin systems integrate Vera CPUs with Rubin GPUs, while NVIDIA also describes CPU-focused systems for agent workloads. The design argument is that tool execution and orchestration need a CPU built and networked as part of the AI estate, not treated as leftover capacity.

The useful distinction is not AMD versus NVIDIA at the first stage. It is dense throughput versus latency-sensitive service work.

Large numbers of independent sandboxes can favour more cores and memory capacity.
Serial orchestration steps, interpreters and database operations can favour stronger per-core performance.
Memory-heavy retrieval and caching can be limited by channels, capacity and locality.
I/O-heavy services need enough PCIe lanes, NIC bandwidth and storage queues.

A mixed CPU estate may be rational. Standardisation is operationally attractive, but forcing every service onto one processor profile can waste power or increase tail latency.

Start with a trace, not a rack count

The right evidence is an end-to-end agent trace. For each task, record time spent in model inference, queueing, orchestration, retrieval, tool launch, tool execution, validation and state storage.

Then capture the resource context:

CPU time and run queue by service;
per-core saturation and frequency behaviour;
memory working set, bandwidth and page faults;
storage IOPS, throughput and p99 latency;
network wait and connection count;
sandbox start time and active sandbox count;
GPU utilisation during CPU-side stages;
completed valid tasks per hour and per kilowatt-hour.

The trace should include failed tasks and retries. An agent that repeats a tool because of a timeout can double CPU and model demand while the business result remains unchanged.

The hardware requirements for agentic AI systems article provides a broader component checklist. This guide focuses on the decision to split CPU services from the accelerator estate.

When separate CPU racks make sense

CPU work delays expensive GPUs

If model workers regularly wait for tool or retrieval stages, and those stages are CPU-bound, dedicated capacity can improve GPU utilisation and task completion. The evidence should show correlation between CPU queues and accelerator idle time.

CPU and GPU services scale differently

An agent platform may add tool workers without changing model capacity, or add a larger model without increasing tool calls. Separate pools let each role scale on its own schedule.

Sandboxes need a stronger isolation boundary

Executing user-generated code beside the model runtime increases risk. Dedicated CPU nodes can limit blast radius, simplify network policy and support different kernel or virtualisation choices.

Large memory pools matter

Retrieval indexes, caches, graph data and active sandboxes can demand more system memory than GPU hosts provide economically. CPU nodes with dense DRAM may be the sensible home for that state.

Failure domains should be independent

A tool-execution surge should not exhaust the host resources needed to serve models. Separate pools let operators reserve capacity, apply different maintenance windows and degrade one service without losing the entire platform.

The facility supports the extra estate

A CPU rack still needs power, cooling, network ports, management, spares and remote hands. It must fit the data-centre plan rather than appearing as a free sidecar to the GPU purchase.

When separate racks are premature

A lightly used internal assistant often has more than enough CPU capacity on its inference server. Adding a separate tier creates extra network hops, deployment pipelines and failure modes before any resource is scarce.

A research team may benefit more from a handful of general-purpose service nodes. They can host Slurm control, repositories, databases and sandboxes without committing to a full rack.

A platform with slow external APIs will not become faster by adding local CPU cores. The trace must separate CPU execution from remote service wait.

Poor software can also mimic hardware shortage. Excessive polling, serial tool calls, repeated parsing and unbounded retries should be fixed before capacity is expanded.

Finally, do not move preprocessing away from GPU hosts automatically. Some pipelines benefit from local data and accelerator affinity. The correct split is determined by data movement and concurrency, not organisational neatness.

The network between CPU and GPU tiers

Once services are separated, every agent step crosses a network boundary. The traffic may include prompts, retrieved context, tool output, embeddings, code artefacts, logs and control messages. Most messages are smaller than distributed-training collectives, but latency and bursts still matter.

The CPU tier does not necessarily need the same fabric as the GPU scale-out network. Management, storage, application access and collectives have different failure and congestion requirements. A sensible design may use:

an accelerated scale-out fabric for GPU collectives;
a low-latency service fabric between CPU and GPU workers;
a storage fabric sized for datasets and checkpoints;
a separate management network.

Some of those can share physical switches with careful isolation. Others warrant separate ports or fabrics. The AI cluster network architecture guide explains how to keep traffic classes and port arithmetic visible.

Measure p99 service latency while the GPU fabric and storage path are busy. A design that works in an empty lab may stall when checkpoints and collectives run together.

Storage and memory can dominate the CPU decision

Agent memory is not one thing. It can include short-lived prompt state, a vector index, database rows, tool artefacts, browser sessions and audit logs. Each has different durability and access requirements.

Keep hot application state near the CPU services that use it. Avoid copying large retrieved documents across tiers repeatedly. Cache deliberately, with limits and invalidation rules. Place persistent data on a storage platform that can survive node replacement and deliver the required metadata performance.

Large DRAM capacity can make a CPU tier attractive for caching and retrieval, but memory must be sized from the working set. Buying the platform maximum because an article lists it is not engineering.

For clusters where storage wait starves accelerators, use the parallel file system buying guide to compare open-source and commercial paths. Agent services may also need transactional or object storage outside that parallel system.

Scheduling is the real architecture

Hardware pools do not create balance by themselves. The scheduler must know which work can run where, how much CPU and memory a sandbox may consume, and what happens when a tier is full.

Set explicit admission controls. Without them, a burst of low-priority agents can consume the tool pool and delay revenue-critical sessions. Reserve capacity for control-plane services so the platform can still recover under load.

Use queue classes tied to business objectives. Interactive coding may require low start latency. Batch document processing may tolerate a queue. Long simulations may need hard CPU and memory limits. These jobs should not all compete on first-come, first-served terms.

The platform also needs backpressure. If CPU workers are saturated, model workers should not keep generating more tool calls that cannot run. A controlled slowdown is better than retries that amplify demand.

An RFQ-ready acceptance test

Ask a supplier or internal platform team to demonstrate the following with the intended agent framework and tools.

1. Replay representative tasks. Include short answers, retrieval, code execution, database access and long multi-step workflows. 2. Measure stage latency. Report p50 and p99 for inference, tool launch, tool execution, retrieval and queueing. 3. Compare three placements. Test services on GPU hosts, on a small CPU pool and on the proposed dedicated tier. 4. Load the system. Increase concurrent agents until one resource becomes the limit. Record where that limit appears. 5. Apply security controls. Run sandboxes with the production isolation, network policy and audit settings enabled. 6. Introduce contention. Run checkpoint or storage traffic and a distributed GPU job while agents use the CPU tier. 7. Fail a CPU node. Confirm task retry, state handling, queue recovery and the effect on model workers. 8. Test upgrades. Roll one service version forward and back without losing the whole agent platform. 9. Report useful output. Count completed valid tasks, not only requests or tokens. Include power and error rate.

The winner should be the smallest architecture that meets the service target with enough recovery margin. A full CPU rack is not a success criterion.

A staged procurement plan

Begin with instrumentation. If traces show CPU-side wait, add a small service tier using the processor profile that matches the work. Keep the network and scheduler design capable of adding nodes.

Move to a dedicated rack only when capacity, isolation or operational ownership requires it. This staged path reduces the risk of fixing ratios before the application has stabilised.

Hosted infrastructure can help during that measurement period. GPUMachines Buy & Host lets a buyer own systems while placing them in a facility with defined power, cooling and remote-hands support. It can be easier to test mixed CPU and GPU estates there than to retrofit an office or small server room.

The GPU Cluster Configurator can then turn the selected ratios into rack units, switch ports, power and cable assumptions.

Questions buyers ask

How many CPU cores are needed per GPU?

There is no reliable universal ratio for agentic AI. GPU host duties, tool concurrency, retrieval, sandboxing and model-serving software all change the requirement. Size each role from traces.

Should agent sandboxes run on GPU servers?

Small trusted workloads can, but untrusted code and high concurrency favour a separate pool. Isolation, kernel choice, memory pressure and failure blast radius should drive the decision.

Is high core count or high frequency more important?

It depends on the service. Many independent sandboxes can use dense cores. Serial orchestration, interpreters and latency-sensitive database work may prefer stronger per-core performance. Mixed tiers are possible.

Can Kubernetes manage both CPU and GPU pools?

Yes, with labels, taints, resource limits and appropriate operators. The difficult work is admission control, locality, observability and version qualification, not creating two node groups.

Do Vera Rubin racks remove the need for external CPUs?

No general conclusion follows. Vera CPUs handle important host and platform roles, but a large agent service may still need separate capacity for tools, retrieval, applications and sandboxes.

Can GPUMachines supply CPU and GPU tiers together?

GPUMachines can design and source mixed compute, networking, storage and hosting configurations, subject to platform compatibility and availability. The proposal should be based on measured service roles rather than a fixed CPU-to-GPU ratio.

Sources and Further Reading

Verdict

Agentic AI makes CPU capacity a first-class design decision, but it does not make a separate CPU rack compulsory. Start by tracing complete tasks. If orchestration, retrieval or tool execution is delaying GPUs, add a CPU pool sized for the specific service. If that pool needs independent scaling, isolation and operations, grow it into a dedicated rack.

Keep the simpler architecture when CPU wait is small, external services dominate latency or the application is still changing quickly. A few well-chosen service nodes can be more useful than a rack purchased from a fashionable ratio.

GPUMachines can help convert task traces into CPU profiles, GPU capacity, fabrics, storage and facility requirements. The goal is not to maximise either processor count. It is to complete useful agent work reliably with no expensive tier waiting on an avoidable queue.

Does an Agentic AI Cluster Need Separate CPU Racks?