🎮 P100 GPU Debugger — CUDA Error Playbook

Everything we know about getting (and keeping) the dual Tesla P100-PCIE-16GB GPUs working for llama.cpp inference on this box. Read the symptom → cause → fix table first, then jump to the section that matches your error.

① 30-second error decoder

Match the exact error string from the container logs — the wording tells you the cause.

② The "unknown error" — dynamic major numbers explained

This is the one that keeps coming back. Understanding it bit by bit:

What is a "major number"?

Every Linux character device (like a GPU node) has two numbers: a major (which driver owns it) and a minor (which specific device). You can see them with ls -l /dev/nvidia* — they appear where a file's size normally would:

crw-rw-rw- 1 root root 506,   0 /dev/nvidia-uvm        ← major 506, minor 0
crw-rw-rw- 1 root root 195,   0 /dev/nvidia0           ← major 195, minor 0
crw-rw-rw- 1 root root 195, 255 /dev/nvidiactl         ← major 195, minor 255

Why 506 one day and 507 the next?

Some NVIDIA devices get a fixed major, others get a dynamic one the kernel assigns from a free pool at load time:

So nvidia-uvm might be 506 today, 507 after the next reboot, back to 506 after another. The exact number is meaningless — what matters is that it changes.

Why that breaks the container

An unprivileged LXC can't touch device nodes unless its cgroup allowlist explicitly permits that major. The config /etc/pve/lxc/102.conf pins them:

lxc.cgroup2.devices.allow: c 195:* rwm     ← nvidia0/1/ctl  (fixed, always right)
lxc.cgroup2.devices.allow: c 507:* rwm     ← nvidia-uvm     (PINNED to 507)
lxc.cgroup2.devices.allow: c 510:* rwm     ← nvidia-caps    (PINNED to 510)
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file

The mount.entry still bind-mounts /dev/nvidia-uvm into the container, so it's visible. But if the real major is now 506 while the cgroup only allows 507, the kernel denies every read/write to it. CUDA tries to open the Unified-Memory device, gets permission denied at the cgroup layer, and reports the generic:

ggml_cuda_init: failed to initialize CUDA: unknown error
warning: no usable GPU found, --gpu-layers option will be ignored

Meanwhile nvidia-smi keeps working because it never touches nvidia-uvm — which is exactly why this masquerades as an app or image bug.

# 1. What are the majors RIGHT NOW?
cat /proc/devices | grep nvidia
#   → e.g. "506 nvidia-uvm"  and  "509 nvidia-caps"

# 2. See what the config currently pins (the stale numbers)
grep cgroup2.devices.allow /etc/pve/lxc/102.conf

# 3. Point the config at the real majors (example: uvm 507→506, caps 510→509)
sed -i 's/allow: c 507:\* rwm/allow: c 506:* rwm/' /etc/pve/lxc/102.conf
sed -i 's/allow: c 510:\* rwm/allow: c 509:* rwm/' /etc/pve/lxc/102.conf

# 4. Restart the container so it re-reads the cgroup allowlist
pct restart 102

Then redeploy the model — you should see both cards in device_info:

CUDA0 : Tesla P100-PCIE-16GB (16269 MiB, 16013 MiB free)
CUDA1 : Tesla P100-PCIE-16GB (16269 MiB, 16013 MiB free)

This is tedious and recurring — automate it. See §⑦.

③ Missing libcuda / "no usable GPU" — driver libraries

Different failure: the LXC has the NVIDIA management lib (so nvidia-smi runs) but not the CUDA driver lib (libcuda.so.1). Symptoms:

• nvidia-smi works but shows CUDA Version: N/A
• llama.cpp: compiled without GPU support even though the image is the CUDA build

The nvidia-container-runtime can only inject libraries that exist on the LXC host, so the fix is to copy the matching driver libs from the Proxmox host into the LXC. Two libs are required:

This is fully scripted in the repo — run it on bigjohnny-100:

./deploy/ml350/fix-lxc-cuda-driver.sh 102

It detects the host driver version, pct pushes both libs into the LXC, creates the libcuda.so.1 / .so symlinks, and runs ldconfig. The libs live in the container's own rootfs, so they survive reboots — but if you upgrade the NVIDIA driver on the host, the versions mismatch and you must re-run it.

④ "no kernel image available" — the sm_60 / arch myth

You'll see advice like "your ARCHS list shows 610 not 600, rebuild llama.cpp for CMAKE_CUDA_ARCHITECTURES=60." On this box that is almost always wrong. How to tell:

P100 = compute capability 6.0 (sm_60), Pascal, no Tensor Cores. Consequence: flash-attention is unsupported — always leave it off for P100 deployments. That's the one arch fact that actually matters here.

⑤ It runs, but it's slow (silent CPU fallback)

If a model deploys "successfully" but generates at ~5–11 t/s, it's probably running on CPU, not the GPUs. Confirm by reading the load log:

docker logs llm-<slug> 2>&1 | grep -iE "device_info|CUDA0|CUDA1|ggml_cuda_init|no usable GPU|offload|buffer size"

• Only CPU in device_info → GPU init failed earlier — scroll up for ggml_cuda_init, then go to §②/§③.
• CUDA0/CUDA1 present but few layers offloaded → model + KV cache didn't fit in 32 GB VRAM and it partially spilled. Lower --ctx-size, drop --parallel, or use a smaller quant. (A 27B Q8_0 at 26.6 GB leaves almost no room for KV — expect trouble.)

Rough healthy numbers on both P100s (≈732 GB/s HBM2 each): a ~16 GB MoE around 60–80 t/s, a 27B dense Q8 around 30–40 t/s. Single-digit t/s = CPU. Use the 🧪 Benchmark Suite (Library → model → Suite) with the GPU presets to measure it properly.

⑥ Copy-paste diagnostic commands

Inside the container (on 102-docker-host)

# does GPU passthrough reach the container?
docker exec -it llm-<slug> nvidia-smi
docker exec -it llm-<slug> ls -l /dev/nvidia*
docker exec -it llm-<slug> sh -c 'echo $NVIDIA_VISIBLE_DEVICES; echo $NVIDIA_DRIVER_CAPABILITIES'

# the actual CUDA error (the line that matters)
docker logs --tail=200 llm-<slug> 2>&1 | grep -iE "cuda|gpu|device_info|CUDA0|CUDA1"

On the LXC host (102-docker-host)

ls -l /dev/nvidia*                 # what majors does the container actually see?
cat /proc/driver/nvidia/version    # driver version

On the Proxmox host (bigjohnny-100)

cat /proc/devices | grep nvidia            # the CURRENT dynamic majors
grep -n nvidia /etc/pve/lxc/102.conf       # what the config pins
nvidia-smi | head -5                        # host driver sanity
systemctl status fix-lxc-nvidia-uvm.service # is the auto-fix healthy?

⑦ Making it permanent — the boot service

Because the uvm/caps majors drift on every reboot, the durable fix is a boot-time service on bigjohnny-100 that re-detects them and patches 102.conf before the container starts. Both files are version-controlled in the repo:

• deploy/ml350/fix-lxc-nvidia-uvm.sh — the patcher
• deploy/ml350/fix-lxc-nvidia-uvm.service — the systemd unit

install -m 755 fix-lxc-nvidia-uvm.sh /usr/local/bin/fix-lxc-nvidia-uvm.sh
install -m 644 fix-lxc-nvidia-uvm.service /etc/systemd/system/fix-lxc-nvidia-uvm.service
systemctl daemon-reload
systemctl enable fix-lxc-nvidia-uvm.service

/usr/local/bin/fix-lxc-nvidia-uvm.sh 102   # prints "Updated ... (uvm=506 caps=509)"
pct restart 102

systemctl restart fix-lxc-nvidia-uvm.service
systemctl status  fix-lxc-nvidia-uvm.service --no-pager

The unit is ordered Before=pve-guests.service, so on a normal reboot the majors are corrected before the container starts and no manual step is needed.