AMD Strix Halo nodes (Linux / Vulkan)

Skulk clusters are not Mac-only. An AMD Ryzen AI Max (Strix Halo, gfx1151) box running Linux can join a cluster as a worker that serves GGUF models on its integrated Radeon GPU, alongside Apple Silicon nodes serving MLX models. The cluster is heterogeneous: each model is placed on the nodes that can actually run it.

An AMD node can run GGUF models through two engines:

• llama_cpp (in-process): the default GGUF path. Skulk loads the model with llama-cpp-python and decodes on the Radeon GPU via Vulkan. Single-node.
• llama_server (served): Skulk launches an external llama-server process and proxies its OpenAI API. This is the only path to llama.cpp's native multi-token prediction (--spec-type draft-mtp), so it is how you get speculative-decoding speedups on an AMD node; see Speculative Decoding for how served native MTP works and its measured gains. Single-node; enabled per node by pointing SKULK_LLAMA_SERVER_BIN at a llama-server binary.

This page covers what such a node needs, how to bring one up, both engines, and how the cluster decides what to run where.

What runs where

A Skulk node advertises the compute backends it can serve, written as <engine>-<compute> tags:

• A macOS node advertises mlx and mlx-metal.
• A Linux node with llama.cpp built for the Radeon GPU advertises llama_cpp and llama_cpp-vulkan.
• If that node also has a llama-server binary (SKULK_LLAMA_SERVER_BIN), it additionally advertises the served backend for its GPU, e.g. llama_server-vulkan (and llama_server-cpu as a floor).

Each model card declares which backends can run it. A GGUF model card lists llama.cpp backends as compatible; an MLX/safetensors model lists MLX. When you launch a model, the master places it only on nodes whose advertised backends include one the card accepts, and prefers the card's higher-ranked backend when several nodes qualify. So a GGUF model lands on the AMD node and an MLX model lands on the Macs, automatically.

The GPU path: Vulkan, not HIP

On gfx1151 the reliable, well-supported way to run llama.cpp on the GPU today is the Vulkan backend (Mesa's RADV driver), not the ROCm/HIP backend. ROCm is not required for inference. It is optional and used only for the rocminfo diagnostic. Skulk's llama.cpp runner offloads through Vulkan (Mesa RADV). On a Ryzen AI Max+ 395 (Radeon 8060S) this fully offloads a 7B Q4_K_M model to the iGPU and decodes at interactive speed, which is what makes the box useful as a cluster node rather than a CPU-only fallback.

Validated configuration

Component	Version
Hardware	AMD Ryzen AI Max+ 395 w/ Radeon 8060S (gfx1151)
OS	Ubuntu 26.04 LTS (kernel 7.x)
GPU compute	Vulkan via Mesa RADV 26.x (Vulkan 1.4, STRIX_HALO)
Build toolchain	cmake 4.x, gcc/g++ 15.x, glslc (shaderc)
Python / uv	uv-managed Python, uv 0.11+
llama-cpp-python	Built from source with the Vulkan backend
llama.cpp	Vulkan build (b9820+), for native MTP (llama-server)

On Ubuntu 26.04 the whole Vulkan path is distro-native (Mesa RADV and vulkan-tools from main, glslc and the optional rocminfo diagnostic from universe, GPU firmware in linux-firmware), so a Skulk node needs no third-party AMD/ROCm apt repository. Inference runs on Vulkan, not HIP: the llama-server binary links libvulkan, not libamdhip, so ROCm is optional and used here only for rocminfo.

Unified memory (kernel parameters)

The Ryzen AI Max has a single pool of unified LPDDR5X. The BIOS carves part of it out as dedicated GPU VRAM (for example 64 GiB on a 128 GiB box) and leaves the rest as system RAM. Two kernel parameters let the GPU address the whole pool, so a model larger than the VRAM carve-out still runs on the GPU (its weights map through the GTT aperture into system RAM):

• amdgpu.gttsize=126976 caps the GPU's unified (GTT) aperture at 124 GiB (126976 MiB / 1024).
• ttm.pages_limit=32505856 raises the pinned-memory limit to the same 124 GiB (32505856 x 4 KiB pages = 126976 MiB).

Set both to fit your box (scale down proportionally on a smaller machine, for example a 64 GiB box). Without them the GPU is capped near the VRAM carve-out and large models fail to load. Skulk already understands this: it shows the machine's full unified memory in the dashboard (VRAM carve-out plus system RAM) and its planner admits models against the whole pool, so a 128 GiB box serves ~128 GiB of model rather than only the non-carve-out slice.

A third parameter is optional and trades IOMMU isolation for throughput and stability on a trusted node:

• amd_iommu=off disables the AMD IOMMU entirely, which removes translation overhead and avoids IOMMU edge cases. Prefer it over iommu=pt only where the node is trusted (the cluster-fabric assumption), since it removes device isolation.

Apply through GRUB and reboot (the parameters live in GRUB_CMDLINE_LINUX):

# swap iommu=pt for amd_iommu=off, keeping the memory caps:
sudo sed -i 's/iommu=pt/amd_iommu=off/' /etc/default/grub
# the line should now read, e.g.:
#   GRUB_CMDLINE_LINUX="amd_iommu=off amdgpu.gttsize=126976 ttm.pages_limit=32505856"
sudo update-grub && sudo reboot
# after reboot, confirm:
cat /proc/cmdline

Bring-up

The full operational steps, a one-shot dependency installer, and a launcher template live in deployment/rocm/ in the repo.

Install the dependencies (one script)

On a fresh Ubuntu box, deployment/rocm/install-deps.sh installs everything the node needs below the Skulk repo (the Vulkan GPU stack, the build toolchain, uv, and GPU device-group membership) and, with flags, builds the MTP llama-server binary and the Skulk uv env in the same pass. It is idempotent:

git clone https://github.com/Foxlight-Foundation/Skulk.git && cd Skulk

# System deps only:
deployment/rocm/install-deps.sh
# Everything, including the MTP llama-server and the Vulkan llama-cpp-python:
deployment/rocm/install-deps.sh --with-llama-server --with-skulk-env
# Verify a box without installing:
deployment/rocm/install-deps.sh --check

If it adds you to the render / video GPU groups for the first time, log out and back in before starting Skulk. The manual steps below are what the script automates.

Manual steps

1. Install the GPU stack: a working Vulkan driver (Mesa RADV); ROCm is optional (only rocminfo). Confirm with vulkaninfo | grep deviceName (expect the Radeon device via RADV) and, if installed, rocminfo | grep gfx (expect gfx1151).
2. Build the Skulk environment: git clone the repo and run uv sync. The Rust networking bindings compile here. No MLX is needed on a non-Mac node.
3. Build llama-cpp-python with Vulkan: the default install is a CPU wheel, so reinstall it built from source with the Vulkan backend. --no-binary forces the source build, otherwise uv installs the prebuilt CPU wheel and CMAKE_ARGS is ignored:

   CMAKE_ARGS="-DGGML_VULKAN=on" uv pip install --force-reinstall \
     --no-cache-dir --no-binary llama-cpp-python \
     --python .venv/bin/python llama-cpp-python

   You build this once. The service entrypoint runs uv sync --inexact on a node that declares a GPU backend, so a routine sync no longer prunes this source-built wheel; rebuild it by hand only when bumping the llama.cpp version. As a safety net, if the wheel is ever replaced by a CPU-only one (no GPU offload compiled in), the node detects that and advertises llama_cpp-cpu instead of its GPU backend, so GPU work is never routed to a degraded build.
4. Build llama-server for native MTP (optional): speculative decoding on the AMD node runs through an external llama-server process (the in-process binding does not expose native MTP). Build it once with Vulkan and point the node at it (see the MTP section below). install-deps.sh --with-llama-server does this for you.
5. Launch the node: point it at the rest of the cluster using the launch-skulk.sh.example template. Skulk detects the Radeon GPU and derives the node's backends automatically, so declaring SKULK_LLAMA_CPP_BACKENDS=vulkan (as the template does) is an explicit override, no longer a requirement; run uv run skulk doctor to audit what the node will advertise. Nodes on the same LAN segment find each other automatically (mDNS); if this node is on a different segment, set SKULK_BOOTSTRAP_PEERS to dial the existing nodes. On Linux there is no launchd, so start it detached so it survives an SSH disconnect:

   setsid bash -c 'exec ~/launch-skulk.sh > ~/skulk.log 2>&1' </dev/null >/dev/null 2>&1 &

A headless node needs no dashboard build: the API serves without the UI, and you reach the dashboard from any node that has it.

Serving a model on the AMD node

Launch a GGUF model (its card lists llama.cpp backends as compatible) and the master places it on the AMD node. From any node's API:

curl -s -X POST http://<any-node>:52415/place_instance \
  -H 'Content-Type: application/json' \
  -d '{"model_id":"<org>/<gguf-repo>","instance_meta":"MlxRing","min_nodes":1}'

Skulk downloads only the preferred quantization (not every quant in a multi-quant repo), loads it through llama.cpp on the Radeon GPU, and serves it through the same OpenAI-compatible endpoints as any other model.

What the AMD node can serve

A GGUF model on the AMD node uses the same OpenAI-compatible endpoints as any other Skulk model, with the same streaming behavior. It matches the MLX nodes on the generation capabilities llama.cpp supports:

• Text generation, streamed token by token.
• Vision / image input for a vision GGUF (with an mmproj projector): send images as OpenAI image_url content and the model describes or reasons over them. The projector loads through llama.cpp's multimodal chat handler.
• Logprobs (opt-in): per-token logprobs and ranked alternatives (logprobs / top_logprobs). This needs the model loaded so it retains per-token logits, which pre-allocates a large buffer (context length x vocab), so it is off by default. Enable it per node with SKULK_LLAMA_CPP_LOGITS_ALL=1; doing so further caps the served context to a bounded window (SKULK_LLAMA_CPP_LOGITS_ALL_N_CTX, default 8192) so the buffer stays small. With it off, a logprobs request returns a clear error rather than silently omitting them. Either way the served context window is sized to the memory the cluster admitted for the instance, never the model's full trained context (a 128k-context model loaded at full context would otherwise allocate a KV cache far larger than placement reserved and could exhaust the node on load).
• Tool calling: a request's tools are passed to the model. Whether the model returns a structured tool call or describes the call in prose depends on the model and its embedded chat template, which Skulk uses as-is; either way the request completes through the normal streaming path.

Vision / multimodal GGUF models are served. A vision GGUF that ships a separate mmproj projector (LLaVA / Qwen-VL style) runs on an AMD node: the projector downloads alongside the weights and the runner loads it through llama.cpp's multimodal chat handler, so image chat requests (OpenAI image_url content) work. A repo is recognized as a vision model from its config.json vision section, or, when it ships none, from the presence of the mmproj file.

Native MTP on the AMD node (served engine)

Speculative decoding (multi-token prediction) does run on an AMD node, through the served llama_server engine. llama.cpp's native MTP lives in the llama-server application, not in the in-process llama-cpp-python binding, so Skulk reaches it by launching llama-server with --spec-type draft-mtp and proxying its OpenAI API. A model whose card declares served_spec_type = draft_mtp and lists llama_server-* backends is placed on a node that has a llama-server binary and served with MTP on.

There are two MTP shapes, both served this way:

• Baked-in heads: a GGUF that ships MTP prediction tensors (for example the Qwen3.5 / Qwen3.6 MTP GGUFs). No separate draft model is needed.
• Draft-model MTP: a base GGUF plus a small separate draft GGUF passed as --model-draft (for example Gemma 4 31B with a published draft). The card co-fetches both through the store.

You no longer need to build llama-server by hand to get this: Skulk's one-command installer (install.sh at the repo root) detects an AMD GPU and installs the prebuilt skulk-llama-server-vulkan engine wheel (a pinned Vulkan llama-server + ggml-rpc-server build), which Skulk's engine provisioning discovers and wires automatically. The same normal install builds the dashboard through Skulk's bundled Node.js runtime even when the Linux host has no system Node/npm; use --headless only for an intentionally API-only worker. A manual build remains a supported override for a custom or newer llama.cpp: point SKULK_LLAMA_SERVER_BIN at your own binary and it always wins over the managed one. Native MTP (--spec-type draft-mtp) landed in llama.cpp build b9196, so use a newer tag:

git clone https://github.com/ggml-org/llama.cpp.git ~/llama.cpp && cd ~/llama.cpp
# -DGGML_RPC=ON also builds ggml-rpc-server, which multi-node GGUF pooling
# needs on donor nodes (see the pooling section below); include it even on a
# single-node build so the node can join a pool later without a rebuild.
cmake -B build -DGGML_VULKAN=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server ggml-rpc-server -j"$(nproc)"
# then in ~/.skulk/skulk.env (or launch-skulk.sh):
#   SKULK_LLAMA_SERVER_BIN=$HOME/llama.cpp/build/bin/llama-server

deployment/rocm/install-deps.sh --with-llama-server runs exactly this and prints the path to set. On a Ryzen AI Max the Vulkan llama-server serves an MTP-capable GGUF on the Radeon GPU and generates with speculation active, streaming reasoning and tool calls through the same /v1 endpoints as any other model. Acceptance is per-model: a pairing that does not pay is turned off in that model's card, the same discipline MLX MTP uses. See Speculative Decoding for how MTP works and what speedups to expect.

Pooling GGUF models across AMD nodes (multi-node)

A GGUF model that fits no single node but fits the combined GPU memory of several AMD nodes can be served by pooling: Skulk places one driver node, which runs llama-server --rpc donor:port,... and holds the model file, and one or more donor nodes, each running a small ggml-rpc-server that lends its GPU memory. llama.cpp splits the weights and KV across the pooled devices itself. Skulk always prefers a single node whenever the model fits one, so pooling only engages for models that genuinely need it (or when a placement requests min_nodes of 2 or more explicitly).

Requirements on every participating node:

• The served engine configured (SKULK_LLAMA_SERVER_BIN pointing at a llama-server built with -DGGML_VULKAN=ON -DGGML_RPC=ON; the install script above does this).
• The ggml-rpc-server binary next to llama-server (the same build produces it), or SKULK_RPC_SERVER_BIN pointing at it explicitly.

Placing a pooled model is the same placement call as any other, with a minimum width:

curl -X POST http://<any-node>:52415/place_instance \
  -H 'Content-Type: application/json' \
  -d '{"model_id": "bartowski/openai_gpt-oss-120b-GGUF",
       "sharding": "Pipeline", "instance_meta": "LlamaRpc",
       "min_nodes": 2, "excluded_nodes": []}'

What to expect, from measurements on a Strix Halo pair:

• Pooling is a capacity feature, not a speedup. Decode runs roughly 15% slower pooled than on a single node that fits the model (the per-token hop to the donors is latency-bound), while prefill can improve slightly because the pooled GPUs share the compute. Use pooling for models a single node cannot hold; keep everything else single-node.
• A USB4/Thunderbolt link between the nodes helps load time, not decode. Donor weight transfer at load rides the link; give the link a static point-to-point subnet (for example a /30 per link with MTU 9000) rather than relying on link-local addressing, which multi-port hosts route asymmetrically. Skulk automatically prefers the Thunderbolt address for donor endpoints when one exists.
• Donors keep a local tensor cache, so reloading the same pooled model is much faster than the first load.
• If a donor dies mid-generation, the in-flight request fails cleanly, the whole pooled instance tears down within seconds, and no orphan processes are left behind.

What is not on the AMD path today

• MTP across the in-process llama_cpp engine. The in-process binding does not expose native MTP; use the served llama_server engine (above) for MTP. A GGUF served through llama_cpp runs plain autoregressive decoding.
• Sharding one model across an AMD node and a Mac. The two engines do not share a runtime; a cycle must agree on one engine. Pooling GGUF models across several AMD nodes IS supported (see the pooling section above).

The interconnect doctrine still applies: the cluster fabric is trusted, so put untrusted segments behind your own network controls.