.jpg)
A private AI cloud is not a GPU cluster with Kubernetes installed. It is a service that lets approved users request accelerators, data access and model endpoints under defined quotas, security controls and support rules. If every job still needs an administrator to choose a node and repair permissions, the organisation owns shared hardware, not a cloud.
Build the service contract first. Decide who can request capacity, which workload classes are offered, how tenants are isolated, where data may live, what the scheduler can pre-empt, how usage is measured and who responds when a job or node fails. Hardware follows those decisions.
The sensible first release is usually a small repeatable block with one or two well-supported service profiles. Measure queue time, GPU occupancy, model-load time, storage pressure and support effort. Add nodes only when the evidence shows what the next block should fix. A large purchase cannot compensate for an undefined platform team or weak access controls.
GPUMachines sells, configures and hosts GPU infrastructure. This guide explains how to frame a private AI cloud project before a bill of materials. It does not replace a security assessment, privacy review, data-protection advice or application architecture. Compare GPU Cloud, Buy & Host and AI Factory designs before assuming the equipment must sit in your own building.
The service has nine parts
A workable private AI cloud joins nine responsibilities:
| Layer | What users should receive | What the platform team must own | | --- | --- | --- | | Service catalogue | Clear GPU and software profiles | Supported images, quotas, limits and lifecycle | | Identity | Approved access linked to real teams | SSO, roles, joiner/mover/leaver process and audit | | Tenancy | Defined sharing and isolation | Namespaces or projects, policies, dedicated pools where required | | Scheduling | Predictable placement and queue behaviour | Priorities, quotas, pre-emption, reservations and fairness | | Compute | Suitable GPU capacity | Node standards, drivers, firmware, health and spares | | Data | Approved datasets and model access | Storage tiers, permissions, encryption, retention and backup | | Networks | Reachable services with bounded traffic | Management, storage, compute and user planes | | Metering | Usage that teams can understand | GPU time, allocation, storage, transfer and showback rules | | Operations | A supported production service | Monitoring, incidents, patching, change control and recovery |
Missing any one of these pushes work onto users. They create private scripts, keep unaudited model copies or reserve GPUs longer than required because the shared service feels unreliable.
Define the users and trust boundary
Internal research teams, regulated business units and external customers do not need the same isolation. Start by naming the tenant types and the consequences of one tenant seeing another's data, memory, logs or network traffic.
For trusted internal teams, “soft tenancy” may use a shared cluster with project namespaces, role-based access, network policies, storage permissions and scheduler quotas. It still needs testing and audit, but the threat model assumes cooperative users and centrally managed workloads.
External customers, contractors or mutually untrusted teams need stronger boundaries. Dedicated node pools, separate clusters, isolated control planes, hardware partitions or separate physical infrastructure may be appropriate. The decision belongs to the security and risk owners, not to a convenience setting in the scheduler.
Kubernetes documentation warns that namespaces alone do not provide every form of tenant isolation. RBAC, NetworkPolicy, resource quotas and data-plane controls must work together; some cases require dedicated clusters or virtual control planes. Write the trust model before selecting the tenancy mechanism.
Identity comes before notebooks
Every person, service account and automation path should have an owner. Integrate the platform with the organisation's identity provider where possible, then map groups to projects and roles. Avoid shared administrator accounts and long-lived credentials stored in notebook files.
Access should follow the least privilege needed for the job. A data scientist may submit jobs and read approved datasets without changing node drivers. A platform engineer may maintain the cluster without reading model-training data. Service accounts need narrower permissions than the people who created them.
NIST SP 800-207's zero-trust model says network location should not create implicit trust. Apply that principle to AI services: authenticate requests, authorise access to resources, protect management endpoints and log policy decisions. Sitting on the office network is not an identity.
Include joiner, mover and leaver processes. Removing a user must revoke interactive access, API tokens, notebook sessions and data permissions. Audit logs need a retention period and an owner who can investigate them.
Build a small service catalogue
Users should request outcomes, not memorise node names. Offer a small number of profiles such as:
- interactive development on one partition or GPU with a time limit;
- batch training on one or several full GPUs;
- reserved production inference with an availability target;
- a dedicated node or pool for workloads that require stronger isolation.
Each profile needs a GPU type, memory allocation, CPU and RAM ratio, storage path, network access, software image, maximum duration, queue policy and support class. State whether jobs can be pre-empted and how far ahead reservations may be made.
Keep the first catalogue narrow. Supporting every framework, driver and container combination creates an unmaintainable service. A curated baseline with a documented exception process gives users freedom without turning each job into a custom cluster.
Scheduler policy is the product
GPU allocation determines whether users trust the platform. A queue that lets one group reserve every GPU indefinitely will drive others back to personal workstations or public cloud accounts.
NVIDIA AI Enterprise includes scheduling and GPU-management components, and NVIDIA's current Run:ai reference material describes central scheduling, quotas and sharing across Kubernetes environments. Whether the project uses Run:ai or another scheduler, specify policy in plain language:
- how quota is divided between teams;
- whether unused entitlement can be borrowed;
- which production jobs receive guaranteed capacity;
- when pre-emption is allowed and how checkpoints protect interrupted work;
- how reservations, priorities and maximum runtimes work;
- what happens when a team exceeds its budget or storage allowance.
Allocation and utilisation are different. A job can reserve eight GPUs while using them poorly. Collect GPU activity, memory use, power, communication and storage wait alongside scheduler records. Show teams idle reservations and stalled jobs before using utilisation numbers for chargeback.
Choose compute pools by workload
One cluster does not need one node type. A private AI cloud can use separate pools with a common access and scheduling layer.
PCIe GPU servers suit independent inference replicas, rendering, virtual workstations and training jobs that split cleanly. They offer broad GPU choice and can run several separate workers in one chassis. Check PCIe lanes, slot spacing, airflow and NIC placement for the exact configuration.
HGX systems suit tightly coupled training and large multi-GPU inference that benefit from NVLink and NVSwitch inside the node. They carry higher power and cooling demands, so the service profile should reserve them for work that can use the architecture.
Workstations can remain part of the service. They are useful for interactive development, visual work and teams that need direct display or peripheral access. Do not force every prototype into the shared cluster if a managed workstation gives a faster and cheaper answer.
Standardise each pool. Fix supported CPU, memory, GPU, NIC, NVMe and firmware profiles; record allowed variations. Repeatable nodes reduce driver drift and make spare planning possible.
Separate the network planes
Draw management, storage, user or API, and east-west compute traffic as separate roles. They may share physical switches in a small pilot, but routing, policy and bandwidth must remain clear.
The management plane carries Kubernetes or scheduler control, provisioning, monitoring and administrative access. Keep BMC interfaces on a restricted out-of-band network. The user plane serves notebooks, APIs and data ingress. Storage traffic moves datasets, checkpoints and models. A distributed training fabric carries collectives between GPU nodes.
Do not let a checkpoint burst starve the control plane. At larger scale, use dedicated fabrics or bandwidth classes and test them together. The RoCE versus InfiniBand guide explains the scale-out decision; choose from workload and operations evidence rather than protocol fashion.
Document egress. Models and datasets can leave through web interfaces, object stores, registries, notebooks and support bundles. Security controls must cover these paths without blocking legitimate work.
Storage is more than a fast shared filesystem
Private AI clouds usually need several storage behaviours:
- home and project space for code and small artefacts;
- high-throughput active data for training;
- low-latency model and container caches;
- checkpoint space with predictable burst writes;
- object or archive storage for retained datasets and model versions;
- protected configuration, metadata and audit logs.
Trying to serve all of them from one tier can create cost or performance problems. Map each data class to its owner, sensitivity, retention, recovery target and access pattern.
Permissions must follow project and tenant boundaries. A scheduler quota does not restrict a shared filesystem unless storage policy agrees. Encrypt sensitive data according to organisational requirements and control the keys separately from ordinary user access.
Benchmark mixed load. Training reads, checkpoint writes, model downloads and metadata operations can collide. Record job step time and recovery time, not just a large sequential throughput number.
Software needs a supported baseline
NVIDIA AI Enterprise describes a stack that includes drivers, GPU operators, vGPU or MIG capabilities, workload management and Kubernetes-related components. The exact stack depends on hardware and licensing, but the principle is useful: drivers, container runtime, orchestration and management tools form one compatibility matrix.
Publish supported software images with versioned CUDA, framework and serving-engine combinations. Scan images, record their source and retire old versions through a stated process. Let teams extend the baseline in project containers without giving them permission to change node drivers.
Air-gapped or restricted environments need an internal path for packages, containers, model weights and security updates. Run:ai documentation describes SaaS, self-hosted and air-gapped deployment modes; select the mode from the security boundary and support plan, not from installation convenience.
Patch in rings. Test a small pool, then move through non-production and production nodes. Keep a rollback path for firmware, driver and scheduler changes. A private cloud that cannot change safely will freeze on an unsupported stack.
Model endpoints are a separate service
Training jobs and production inference have different operating needs. A model endpoint needs versioned deployment, health checks, scaling policy, request limits, logs, rollback and an availability target. A notebook that exposes a port is not a production endpoint.
Decide who approves models for serving, how artefacts are signed or registered, and how data reaches the endpoint. Record the model licence and any restrictions on commercial use. Monitor queue time, token or image latency, error rate, GPU memory and model-load failures.
Production services may deserve reserved GPUs or a dedicated pool so that an urgent training run cannot evict them. If the service must survive a node failure, prove that replicas restart and reload within the recovery objective.
Metering, showback and chargeback
Users need to see what they consume. Start with showback: report allocated GPU-hours, active GPU time, peak memory, storage use, reservations and queue delay by project. Explain the difference between reservation and useful work.
Chargeback adds policy and finance. It must account for reserved production capacity, shared platform cost, storage, data transfer, support and idle headroom; a simple public-cloud hourly comparison can mislead. Agree the method with finance and service owners before sending bills to teams.
Metering also informs capacity. If queues are long but GPU activity remains low, adding hardware may reinforce a software or data bottleneck. If a pool stays busy with well-used jobs and predictable demand, the case for another repeatable block is stronger.
Security and data governance
Classify data and models before they enter the service. Some projects may allow ordinary internal storage, while regulated or customer data needs tighter access, retention and location controls. Define who can bring external models or containers into the environment and who reviews their licences and security.
Protect the supply chain: use trusted registries, vulnerability scanning, signed artefacts where supported and controlled promotion between development and production. Keep management interfaces away from public access. Segment BMC, orchestration, storage and tenant traffic.
Incident response must include GPU-specific evidence. Preserve scheduler events, container logs, node health, network telemetry, model and image versions, identity records and storage access. Decide how the team isolates a node or tenant without destroying the information needed to investigate.
Zero trust does not mean inserting an authentication proxy and declaring success. It means each request receives an identity and policy decision appropriate to the resource, while monitoring checks whether the controls still work.
On premises, hosted or mixed
On-premise deployment offers physical control and can keep data close to existing systems. It also places power, cooling, hardware service and platform staffing on the buyer. Confirm the rack envelope before selecting dense GPUs.
A hosted private cloud can preserve dedicated hardware while moving facility work to a suitable data centre. GPUMachines Buy & Host is one route to compare. Define access, remote hands, data transfer, incident responsibility and exit arrangements in the design.
A mixed model can keep sensitive or steady workloads on private capacity and use external cloud for bursts or experiments. The difficulty is portability: identity, data paths, images, observability and cost reporting must work across locations. Test movement with a real workload before promising hybrid operation.
Build in four releases
Release 1: measured pilot
Start with a small node pool, identity integration, one storage path, basic scheduling and two service profiles. Select a few real users. Measure demand, failures and support tickets for long enough to expose routine behaviour.
Release 2: governed shared service
Add project onboarding, quotas, network policy, approved images, monitoring, backup, showback and a defined support rota. Fix the problems that drove users around the pilot.
Release 3: production services
Create reserved inference or training pools, model deployment controls, stronger recovery, maintenance rings and formal service targets. Run failure and restore tests.
Release 4: repeatable expansion
Add node blocks, switch capacity and storage according to measured bottlenecks. Preserve the standard configurations and topology. Revisit tenancy if external or less-trusted users enter the platform.
Acceptance tests
Approve the service, not just the hardware. A useful acceptance plan covers:
1. A new user gains only the approved project access. 2. Quotas and priorities behave under competing workloads. 3. A job can checkpoint, stop and resume on another healthy node. 4. Storage meets agreed read, metadata and checkpoint targets under mixed load. 5. Network policies block prohibited tenant paths while allowed services remain reachable. 6. A failed GPU node is drained, repaired and returned without hidden configuration drift. 7. A production model endpoint survives the agreed failure case. 8. Usage reports reconcile with scheduler and infrastructure telemetry.
Include a leaver test and an incident-evidence test. Access that cannot be revoked and logs that cannot answer who did what are platform defects.
When not to build a private AI cloud
Do not build one for a single short experiment, an undefined model plan or a team with no platform owner. A managed workstation, GPU Cloud or hosted server will reach useful work sooner.
Pause if the facility cannot support the power and cooling envelope, if security ownership is unresolved, or if the organisation cannot commit staff to updates and incidents. Buying more GPUs creates more unsupported surface area.
And do not copy a hyperscaler's service catalogue. A private platform should solve the organisation's repeated needs with a small set of dependable profiles. Breadth can come later.
FAQ
Is Kubernetes enough for a private AI cloud?
No. Kubernetes provides orchestration primitives, but the service still needs identity, tenancy controls, GPU scheduling, storage, approved software, metering, support and lifecycle policy.
Can namespaces isolate tenants?
Namespaces help organise and scope access, but Kubernetes documentation says multi-tenancy needs additional controls such as RBAC, NetworkPolicy and quotas. Untrusted tenants may require dedicated clusters, control planes or hardware.
Do we need HGX servers?
Only for workloads that benefit from their tightly connected multi-GPU architecture. Independent inference and many development jobs may fit PCIe servers or workstations at lower cost and facility demand.
How should teams pay for shared GPUs?
Start with transparent showback. Report allocation, active use, reservations, storage and queue behaviour. Add chargeback only after finance and service owners agree how shared capacity and support are valued.
What should the first purchase include?
A small repeatable compute block, suitable storage and networking, management capacity, spares and the software or support needed to operate two or three defined services. Leave room to learn before scaling.
Verdict
The hard part of a private AI cloud is not installing GPUs. It is turning shared infrastructure into a service with predictable access, isolation, scheduling, data paths, usage records and support.
Define that service, prove it on a small block and expand from measured demand. Use the GPU cluster configurator to model the hardware, then compare on-premise, hosted ownership and AI Factory routes before fixing the location.