Building a Dedicated AI/ML Rig

I built this rig to have a dedicated environment for AI/ML work — training models, experimenting with large datasets, and running GPU-accelerated workloads. Right now I’m working through Transformers for Natural Language Processing by Denis Rothman on it.

The Build

NVIDIA Drivers

Add the repo and install the drivers + CUDA. Blackwell cards like the 5090 need driver 570 or newer and NVIDIA’s open kernel modules — the proprietary latest-dkms stream won’t drive them.

1
2
3
4
sudo dnf config-manager --add-repo https://developer.download.nvidia.com/compute/cuda/repos/rhel9/x86_64/cuda-rhel9.repo
sudo dnf module install nvidia-driver:open-dkms -y
sudo dnf install cuda-toolkit -y
sudo reboot
  • Verify: nvidia-smi should show RTX 5090.
  • Pin driver if needed: sudo dnf module install nvidia-driver:570-open.

Headless GPU Setup (No Display)

If your server doesn’t have a monitor attached (like mine), you’ll want to blacklist the nouveau driver and configure NVIDIA for headless operation:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# Blacklist the nouveau open-source driver
echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf
echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.conf

# Disable the NVIDIA framebuffer (not needed without a display)
echo 'options nvidia-drm fbdev=0' >> /etc/modprobe.d/nvidia-headless.conf

# Rebuild initramfs and reboot
dracut -f
reboot

After reboot:

1
2
3
4
5
# Enable persistence mode (keeps the GPU initialized, faster job starts)
nvidia-smi -pm 1

# Verify it's running headless
nvidia-smi -q | grep "GPU Operation Mode"

Storage

The 6x WD Red 3TB drives run on ZFS RAIDZ1, giving about 15TB of usable storage for datasets and models. The PCIe SATA card (ASM1166) handles all six drives.

Gotchas

PCIe lane saturation: The i9-14900K only has 20 PCIe lanes. The RTX 5090 takes 16 (x16 slot), which leaves just 4 lanes for everything else — NVMe, SATA controller, 10Gb NIC. In my case, I can’t run all the drives and the 10Gb NIC at the same time. Something to keep in mind if you’re planning a similar build — you may need to prioritize or look at a platform with more lanes (like Xeon or Threadripper).

Performance Tips

  • CPU Governor: echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
  • NVMe Scheduler: echo none | sudo tee /sys/block/nvme0n1/queue/scheduler

Remote Management

The Geekworm KVM-A8 with a Raspberry Pi 4 gives me full remote KVM over IP — BIOS access, OS installs, troubleshooting, all without a monitor. Essential for a headless rig in a rack.

What’s Next

With the hardware sorted, the next step was getting JupyterLab installed and configured. I covered that in a separate post.