GPU Server SSH, DNS and Time Synchronisation

Create named admin accounts, test SSH keys from a second host, and verify DNS and time synchronisation before installing GPU drivers.

A GPU node can pass POST and still be unfit for a cluster. If its hostname changes, its administrator account is shared, its SSH host key was cloned, or its clock drifts, later failures become difficult to separate from driver, scheduler and fabric problems.

Set identity and access before the NVIDIA software stack. The practical order is: record the intended name and addresses, create named accounts, prove key-based SSH from a second machine, check DNS in both directions, confirm time synchronisation, then capture the evidence. Keep the BMC or physical console available until every remote-access change has passed.

This is a source-backed commissioning guide for Ubuntu Server. It is not evidence that GPUMachines ran these commands on your system; test each change against your Ubuntu release, identity service and network policy.

Write the node identity record first

Decide what the node will be called before configuring SSH or joining it to a scheduler. A short hostname such as gpu-a17 is useful on the console, while a fully qualified domain name such as gpu-a17.compute.example.com gives DNS an unambiguous target.

Record the following for each node:

| Item | Example | Acceptance evidence | |---|---|---| | Short hostname | gpu-a17 | hostname --short | | FQDN | gpu-a17.compute.example.com | hostnamectl --static and forward DNS answer | | Management IP | 10.20.4.17 | reachable only from the management path | | BMC address | 10.10.4.17 | separate inventory record | | DNS servers and search domain | site-specific | resolvectl status | | Time source | two approved NTP endpoints | timedatectl or chronyc evidence | | Admin group | gpu-admins | getent group gpu-admins | | SSH host-key fingerprint | SHA256 fingerprint | checked out of band |

Set the hostname with the operating system's supported interface:

``bash sudo hostnamectl set-hostname gpu-a17.compute.example.com hostnamectl status hostname --short hostname --fqdn ``

This example stores the FQDN as the static hostname. hostnamectl --static should therefore return gpu-a17.compute.example.com, while hostname --short returns gpu-a17. The hostname --fqdn result also depends on the resolver and host records; setting the hostname does not create DNS records.

Do not use a reused VM or installer image as proof that identity is correct. The hostname, machine identity and SSH host keys all need to belong to the commissioned node. A golden image must generate unique host keys on first boot rather than distributing one private host key to a fleet.

For the installation steps that precede this work, see Installing Ubuntu on a GPU Server. If the management network is not already isolated, fix that before exposing SSH; the IPMI and BMC guide explains why host and out-of-band access should not share an unrestricted path.

Give people named accounts, not a shared login

A shared admin account weakens attribution and makes offboarding awkward. Give each operator a named identity. Keep automation identities separate from human accounts, restrict their privileges to the job they perform, and use a non-interactive shell when a service never needs a terminal.

On a standalone Ubuntu node, the basic local-account path is:

``bash sudo adduser alice sudo addgroup gpu-admins sudo adduser alice gpu-admins sudo adduser alice sudo id alice sudo -l -U alice ``

Ubuntu disables the root password by default and expects administrators to elevate through sudo. SSH root access also depends on the effective PermitRootLogin policy, so check it with sshd -T rather than assuming. Named elevation produces a clearer audit trail than distributing a root password. At fleet scale, connect the node to the organisation's approved identity service instead of reproducing local accounts across every server; the local recovery account should remain tightly controlled and documented.

Home-directory permissions also matter. Recent Ubuntu releases create private home directories by default, but an upgraded or restored system may carry older permissions. Check rather than assume:

``bash ls -ld /home/alice getent passwd alice getent group gpu-admins ``

If a person leaves, disable access through the authoritative identity system and revoke their keys. Do not recycle their numeric UID for another user while files with that ownership remain on shared storage.

Stage SSH keys before disabling passwords

Install the server component if the Ubuntu installation did not include it:

``bash sudo apt update sudo apt install openssh-server sudo systemctl enable --now ssh.service sudo ss -lntp | grep ':22' ``

Verify the server's SSH identity out of band

The first SSH connection asks the client to trust a host-key fingerprint. That prompt is not proof by itself; it only tells you that the client has not recorded this host key under that name before.

Read the Ed25519 host-key fingerprint from the server console or another trusted management channel:

``bash sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub ``

Compare the SHA256 fingerprint with the client prompt, then record it in the commissioning evidence or a managed SSH host certificate system. If a known-host warning appears later, stop and identify the cause. A legitimate rebuild, a DNS mistake and an interception attempt can present the same warning to the user.

At larger scale, an SSH certificate authority removes much of the manual host-key distribution work, but it adds a signing service and certificate-lifecycle responsibility. Do it as a fleet design, not as an ad hoc fix on one node.

Generate a key on the administrator's trusted client, not on the server on their behalf. Ubuntu recommends Ed25519 for ordinary SSH keys:

``bash ssh-keygen -t ed25519 ssh-copy-id alice@gpu-a17.compute.example.com ssh -S none -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 \ -o PreferredAuthentications=publickey alice@gpu-a17.compute.example.com ``

A key needs a passphrase unless an approved protected automation system holds it. ssh-copy-id may use the existing approved enrolment method to install the public key. The following key-only test must not fall back to the account password: a private-key passphrase prompt is acceptable, but an account-password prompt means the test has failed. On the server, the account should own its .ssh directory and authorized_keys; no other user should be able to write the key file.

``bash sudo -u alice stat -c '%U %G %a %n' /home/alice/.ssh /home/alice/.ssh/authorized_keys sudo -u alice chmod 700 /home/alice/.ssh sudo -u alice chmod 600 /home/alice/.ssh/authorized_keys ``

Open a second terminal and run the explicit -S none key-only test before changing password policy. This prevents SSH from reusing a ControlMaster connection. Keep the first session open. If this is a remote commissioning job, keep the BMC console connected as the independent recovery route.

Apply a small SSH policy and inspect the effective result

Ubuntu supports drop-in files under /etc/ssh/sshd_config.d/. Its main configuration includes that directory near the top, and OpenSSH generally uses the first value it reads for a directive. File ordering therefore matters. Inspect existing snippets before adding another:

``bash sudo grep -RHE '^(Include|PermitRootLogin|PubkeyAuthentication|PasswordAuthentication|KbdInteractiveAuthentication|AllowGroups)' \ /etc/ssh/sshd_config /etc/ssh/sshd_config.d/*.conf 2>/dev/null ``

Before editing, record whether the proposed drop-in already exists and save any existing version under a unique backup name outside /etc/ssh/sshd_config.d/. Keep a copy of the main configuration and any other snippets you change. A backup left in an included directory with a .conf suffix can still affect the effective policy.

A conservative key-only baseline, after key access has passed, can be stored in an early drop-in such as /etc/ssh/sshd_config.d/10-gpu-baseline.conf:

``text PermitRootLogin no PubkeyAuthentication yes PasswordAuthentication no KbdInteractiveAuthentication no ``

Do not paste an AllowGroups rule until every required human, service and break-glass identity has been inventoried. The same caution applies to disabling agent, TCP or X11 forwarding: those controls may be sound, but they can also break remote development, controlled file transfer or orchestration workflows. Decide by server role.

Use KbdInteractiveAuthentication no only where the approved authentication policy permits key-only access. It can break PAM or OTP flows. Test the effective policy for every required identity and network path because Match rules can change the result.

First, validate syntax by itself:

``bash sudo sshd -t ``

Stop unless that command exits with status zero. Then inspect both the general settings and the connection-specific result. Replace the angle-bracket values with the real jump-host and server addresses:

Only restart after both outputs match the approved policy:

``bash sudo systemctl restart ssh.service sudo systemctl --no-pager --full status ssh.service ``

Now repeat the explicit ssh -S none key-only test from the second terminal and run a harmless sudo command. Only close the original session when the new connection, name resolution and elevation path all work.

If the new connection fails, use the BMC or physical console to restore the previous version of every file you changed. If you created 10-gpu-baseline.conf for this change, move it outside the included directory rather than leaving it active beside the restored main file. Run sudo sshd -t again; restart ssh.service only after the check exits zero. Then repeat the new key-only connection and sudo test before continuing the rollout.

Make forward and reverse DNS agree

A GPU scheduler, monitoring platform, certificate workflow and log store can each interpret names differently. Pick one canonical FQDN and make the record consistent. Configure the site's approved DNS servers, search domains and records through its supported network and DNS management tools first; the checks below validate that configuration. From a second host on the same management network, check the address record and reverse mapping:

``bash getent ahosts gpu-a17.compute.example.com dig +short A gpu-a17.compute.example.com dig +short AAAA gpu-a17.compute.example.com dig +short -x 10.20.4.17 ``

The IPv6 query may correctly return no result on an IPv4-only management network. Record that as an intentional state rather than manufacturing an AAAA record. Reverse DNS should return the canonical name where the organisation's operating model depends on it.

On the node, inspect the resolver path:

``bash resolvectl status resolvectl query gpu-a17.compute.example.com readlink -f /etc/resolv.conf getent hosts gpu-a17.compute.example.com ``

systemd-resolved can learn servers and search domains from global configuration, per-link configuration, DHCP or a network manager. Do not overwrite /etc/resolv.conf until you know who owns it. On a standard resolved setup it is usually a symlink to a generated file, so a manual edit either disappears or bypasses per-interface routing.

Split-DNS designs deserve an extra test. Query the node name from the management subnet, a login node and an ordinary corporate client. A record that works only on the server itself is not accepted.

Use one time-sync service and prove that it is locked

Correct time is part of identity. Certificate validation, Kerberos, distributed logs and scheduled work all become unreliable when nodes disagree about the clock. GPU collectives may not fail because of a few milliseconds of offset, but the event trail used to diagnose them can become useless.

Configure approved time sources through the site's supported build or configuration-management tooling before running these acceptance checks. Identify the installed service before testing it:

``bash systemctl is-active chrony.service systemctl is-active systemd-timesyncd.service ``

Chrony is the default time service from Ubuntu 25.10. On current installations, check it first:

``bash chronyc tracking chronyc sources -v systemctl --no-pager --full status chrony.service ``

The selected source should be marked in chronyc sources -v, while chronyc tracking should report a normal leap status and acceptable stratum and offset for the site's policy. Retain the selected source and measured offset with the node record.

Ubuntu's default Chrony configuration uses Network Time Security (NTS). When using those sources, the firewall must permit outbound TCP 4460 for NTS key exchange as well as UDP 123 for time traffic to the approved endpoints. Allowing UDP 123 alone can leave the default NTS configuration unable to synchronise. Inspect the configured source files and the site's access policy before changing either.

Earlier releases and upgraded systems may use systemd-timesyncd. If that is the active service, check it with:

``bash timedatectl status timedatectl timesync-status systemctl --no-pager --full status systemd-timesyncd.service ``

The acceptance signal is not merely NTP service: active. Confirm that System clock synchronized reports yes, and record the selected server, stratum, offset and recent packet count.

Run one time synchronisation service. The commissioning record should show which daemon owns the clock. Use two or more approved sources that represent the same organisational time policy; do not mix an internal isolated source with unrelated public pools without understanding the failure behaviour.

Keep servers on UTC unless an application has a documented reason to use another zone. Operators can convert timestamps in their tools, while UTC keeps cross-node logs comparable.

Roll out access changes as a canary

Do not change an entire GPU fleet from one terminal. Apply the configuration to one non-production or drained node, then repeat the test from the same jump host and network path used by operators.

A sensible run order is:

1. Confirm BMC or physical console access. 2. Capture current SSH, resolver and time configuration. 3. Add the named user and public key. 4. Test a new SSH session and sudo elevation. 5. Validate sshd -t, apply the SSH policy, and test again. 6. Check forward DNS, reverse DNS and the resolver's active servers. 7. Prove time synchronisation and compare the node with the reference source. 8. Store the evidence, then move to the next node through configuration management.

When access and identity pass, continue with the NVIDIA driver, CUDA and Fabric Manager compatibility guide. That later work is easier to diagnose because the host already has a stable name, attributable administrators and trustworthy timestamps.

Acceptance record for every GPU node

Use a pass/fail record, not a screenshot collection without context.

| Check | Pass condition | Evidence to retain | |---|---|---| | Host identity | short name and FQDN match the build record | hostnamectl --static, hostname --short and forward DNS output | | Forward DNS | canonical FQDN returns intended address | dig output from a second host | | Reverse DNS | address returns intended FQDN when required | reverse query output | | Resolver ownership | DNS servers and domains come from the intended interface/configuration | resolvectl status | | Named access | each administrator uses an individual account | id and identity-system record | | Public-key login | new session succeeds from approved jump host | client verbose log or session record | | Root policy | effective SSH configuration blocks direct root login | selected lines from sshd -T | | Password policy | effective state matches the site's approved authentication method | sshd -T output | | Host identity check | fingerprint matches trusted inventory | SHA256 fingerprint | | Time | clock reports synchronised to an approved source | timedatectl timesync-status or chronyc tracking | | Recovery | console path works independently of host SSH | commissioning sign-off |

Record the command, timestamp, node name and result together. Without that context, two identical outputs copied from different hosts are easy to confuse.

Common failures and what they usually mean

Key login works until the policy file is added. Check the effective configuration with sshd -T, the ordering of drop-ins, account group membership and the ownership modes on .ssh and authorized_keys.

The FQDN resolves on the node but not on the jump host. Local /etc/hosts, split DNS or a different search domain is hiding the real record. Test the full name through the same resolver path the operator uses.

The reverse lookup returns an old hostname. The address was reused without updating its PTR record. Fix DNS ownership; do not add more local aliases to conceal it.

timedatectl says NTP is active but the clock is not synchronised. Inspect reachability to the configured sources, service logs and source selection. An enabled service is not an accepted clock.

A rebuild triggers host-key warnings. Verify the rebuild and new fingerprint through the console before removing the old known-host entry. Never train operators to delete warnings automatically.

Password login is disabled and automation stops. The automation path was never included in the identity inventory. Restore through the console, define a dedicated non-human identity and test its exact command scope before hardening again.

Questions operators ask

Should SSH listen on a non-standard port?

Changing the port reduces background scans but does not replace network restriction, key authentication, patching or monitoring. Keep the service on a management network or approved access gateway, then use the port your operating model can support consistently.

Should every administrator have sudo access?

No. Give full sudo only to people whose role requires it. Use narrower command rules or application-specific groups where they are practical, and review them after staff or service changes.

Is a hosts-file entry enough for a small cluster?

It can prove a temporary build, but it becomes a separate database on every machine. Authoritative DNS or managed host records are easier to audit once more than a few nodes, jump hosts and services depend on the names.

Do we need Chrony instead of timesyncd?

Chrony is the Ubuntu default from 25.10. Earlier releases or upgraded systems may still use systemd-timesyncd. Use the daemon selected by the site's supported build, run only one time service, and record which service owns the clock.

Can we disable password authentication during installation?

Yes, if the public-key path and recovery console have already passed. Doing it earlier turns a small key-placement mistake into a lockout.

Sources and further reading

> Commission the management path before the software stack: configure a GIGABYTE G593-ZD1-AAX3 GPU server with GPUMachines and include access, naming and time acceptance in the delivery plan.

Editorial owner: GPUMachines.

Cover image: GIGABYTE G593-ZD1-AAX3 GPU server, from the GPUMachines product catalogue. The photograph illustrates the server platform; it does not document testing of this guide.

GPU Compute
Machines

Configuring SSH, Users, DNS and Time Synchronisation for a GPU Environment