An HPC platform researchers actually use

My first GPU platform, on Kubeflow, was barely used. I asked researchers why, moved to Slurm, and built a cluster that puts every GPU to work.

Role: MLOps platform architect
Built for: The research teams
Duration: 2025 – 2026
Stack: Slurm · NVIDIA Container Toolkit · Ansible

13 nodes
1 login node, 2 masters and 10 GPU compute nodes
All GPUs
shared through one queue, not only the fastest
1 stack
for every team, instead of one per researcher

01 · The need

The GPUs were there, but used badly. Researchers queued up for the most powerful ones and left the others idle. They logged into the machines directly over SSH and each installed their own package versions, so we had no control over the environment and bugs came back all the time. There was no real automated MLOps pipeline. My first answer was a platform on Kubeflow: it went live, but adoption stayed far below the target. I interviewed the researchers, and the answer was clear: Kubeflow was too complex for their daily work.

02 · Constraints and decision

Constraints: Users · researchers, not platform engineers; the workflow had to stay familiar.
Hardware · the GPU nodes already in place.
Environment · one controlled software stack, no more installs by hand on the nodes.
Options: Keep Kubeflow · train every team on a powerful platform, but it asked researchers to change how they work.
Back to direct SSH · add rules, so nothing changes for users, and nothing is under control either.
Move to Slurm · a simpler scheduler that researchers already knew from research computing, and that hands out every GPU through a queue.
Decision: Delivery · a POC, then an MVP, then production, with the researchers testing each step.
Scheduling · Slurm with 2 masters, 1 login node and 10 GPU compute nodes, split into training, processing and interactive partitions.
Environment · jobs run in containers with the NVIDIA Container Toolkit, so every team gets the same stack and versions.
Access · researchers keep VS Code, Jupyter and SSH with single sign-on, and submit from the login node instead of logging into the GPUs.
Operations · nodes deployed with Ansible, authenticated with Munge, a shared NAS for data, and Grafana to monitor the cluster.

03 · Architecture

Data scientists connect from their code editor over SSH to a Slurm login node; the Slurm master schedules jobs on three partitions (training, processing, interactive), records them in the Slurm database, and the compute nodes share a NAS; Grafana monitors the cluster. — Three partitions for three kinds of work, one shared storage, one monitoring view. Click to enlarge.

Components per node: VS Code, Jupyter and SSH on the data scientist's computer with SSO; Slurm CLI, Docker Compose and Munge on the login node; slurmctld, job configuration and Munge on the master; the Slurm daemon and phpMyAdmin on the database node; the Slurm daemon, Munge and CPU and GPU on the compute nodes; a NAS for data. — The tools researchers already use on their laptop, single sign-on, and Munge to authenticate every node. Click to enlarge.

Slurm
NVIDIA Container Toolkit
Ansible
Docker
Grafana

04 · Result

Adoption · from a platform barely used to one several research teams use every day, with most training jobs running on it.
Every GPU is in use · jobs go to the queue, not to the fastest machine.
One controlled environment · no more package versions installed by hand on the nodes.
No new habits to learn · researchers kept the tools they already used.

05 · What I learned

Adoption over launch dates · the first version was live, and still a failure until I asked users why.
Adoption comes from habits · researchers use a platform that fits their tools, not one that asks them to change.
Environment first · most of the bugs came from the setup, not from the code.

Next case study

Engineering teams · 2026Agentic AI that engineers trust

Open to AI Platform, DevOps and Forward Deployed Engineer rolesLet’s build what runs next.contact@stevenoumi.com