Skulk Model Store

The model store is one of Skulk's core multi-node features.

In a normal cluster without a model store, each node may need to download model data for itself. With the model store enabled, one node becomes the shared store host and other nodes stage from it over the LAN.

Why You Would Use It

Use the model store when:

• you have more than one node
• your models are large
• you want fewer repeated downloads
• you want a cleaner offline story after the first download
• you want model files to live on a dedicated large disk or volume

What Changes When It Is Enabled

Without the model store:

• nodes download model data independently
• cold starts can be slower across the cluster
• repeated downloads are more common

With the model store:

• one node hosts the shared model store
• other nodes stage needed files from that host
• Skulk keeps the same cluster and inference architecture, but changes where model artifacts come from

Installed Cards and Existing Caches

Every complete canonical or staged artifact carries an atomic .skulk/installed-card.json sidecar. The versioned record retains the full model card, immutable registry identity and provenance when available, exact artifact selection, base or companion role, owning base card, and a canonical SHA-256 file manifest. The sidecar is durable truth; registry.json is a rebuildable index. The dashboard labels the immediately placeable group Fits this cluster. This is a live capacity statement, not a recommendation or qualification claim. Registry-backed rows also show the signed origin as Foxlight, Agent, or Community; provenance describes authorship, not runtime confidence. When a model root is mounted read-only, Skulk stores the same strict record under its data directory, keyed by the resolved artifact path and manifest digest. This fallback changes only metadata placement; the model bytes remain untouched in their mounted root.

On startup, Skulk reads installed cards before contacting the registry. In SKULK_OFFLINE=true mode, an installed model remains usable indefinitely while its manifest is complete; the registry client's bounded last-known-good window does not expire installed artifacts. Custom cards retain precedence. If the registry removes or replaces a card, the installed generation remains active and the dashboard reports installed_not_current or update_available until a complete replacement commits.

Pre-existing caches are associated only with trusted card sources: an existing sidecar, persisted download state, custom card, bundled card, or signed catalog. A matching directory name never creates signed verification by itself. A full immutable revision marker plus matching artifact selection is registry_verified; complete bytes without that proof are local_legacy and remain usable under their retained effective card. Unmatched directories are inventoried as unresolved and are not launched or imported automatically. Interrupted or partial directories never receive a legacy installed-card sidecar merely because their directory name matches a trusted card.

Automatic Reconciliation

The authoritative store host periodically inventories bounded node-local cache summaries outside replicated State and the event log. Replicas deduplicate by installed identity and manifest digest. When the store is missing an artifact, it prefers a same-host copy, then a revision-verified copy, then a deterministic healthy source node.

The source issues a random short-lived capability bound to the source, target store node, manifest digest, byte ceiling, and expiry. The store pulls files with HTTP ranges into a resumable temporary generation, verifies every size and SHA-256 digest, writes the sidecar, and atomically publishes the generation. Failed replacement transfers leave the previous generation intact. Source node caches are never removed during migration. Capacity admission credits bytes already retained in valid partial files, while the export capability enforces its manifest-bound cumulative byte ceiling and rejects source files changed after issuance. The internal store import mutation accepts only a direct loopback peer and rejects proxy-forwarding headers, so a local reverse proxy cannot turn it into a remote mutation surface.

Automatic imports are enabled by default. On a production fleet where you want to inspect the first migration before moving bytes, temporarily enable inventory-only mode:

model_store:
  reconciliation:
    enabled: true
    inventory_only: true
    interval_seconds: 300

Inspect GET /store/reconciliation, then restore inventory_only: false to resume automatic imports. POST /store/reconciliation/rescan is a loopback-only operator retry; periodic reconciliation remains the normal path. Inventory and capability-bound export cover the staging cache, direct-download fallbacks in SKULK_MODELS_DIR, and configured read-only model roots. A canonical index entry suppresses import only while its adjacent sidecar and complete manifest still validate.

An operator store deletion writes a durable alias tombstone before removing the canonical generation. Reconciliation continues to report any node caches that missed the best-effort eviction, but it will not import the deleted base artifact or companions owned by that base card. The tombstone remains through restarts; a later explicit, successfully completed store download clears it.

GGUF repositories download only the pinned quantization

A GGUF repository often ships several quantizations of the same model (for example Q4_K_M, Q5_K_M, Q8_0, bf16). The store downloads only the quantization a model card pins (its gguf_file), plus the multimodal projector for a vision model, rather than every quant in the repository. This keeps a single-quant download to roughly the size of that one file instead of the whole repo.

Qualified cards can pin immutable artifacts

A model card may set source_revision to a full Hugging Face commit hash. The store records both that revision and the effective source repository with its registry entry. Either mismatch identifies a different artifact generation, even when the selectable alias is unchanged. If either identity field changes, Skulk downloads and registers the replacement before removing the previous canonical copy; concurrent downloads deduplicate only when both fields match.

Worker staging enforces the same discipline: each staged directory records the revision it was staged from, and a staging request checks that record before reusing anything. A staged directory recorded at a different revision is deleted and re-staged from the pinned artifact; it is never treated as a cache hit, so a pin change can never silently serve the previous commit's weights. On a store host whose staging path is the store itself, a pinned model resolves directly to its revision-qualified canonical store directory.

Cards without source_revision retain the historical behavior and follow the repository's mutable main branch.

The store host advertises a routable address

The store host broadcasts the address other nodes use to reach it. Even when you configure store_host as a hostname, the store host resolves and advertises its own best routable IPv4 (a private LAN address is preferred). This avoids a failure mode on a Thunderbolt-meshed fleet, where a bare hostname could resolve through mDNS to a link-local Thunderbolt address (169.254.x) that a peer without a direct Thunderbolt link cannot reach, even though the LAN path works. An operator-supplied routable IP in store_http_host is still honored as-is.

What Does Not Change

• the libp2p mesh, election, master, and worker model stay the same
• the main Skulk API stays the same
• the dashboard remains your main control surface
• single-node Skulk still works fine without the model store

Before You Start

Make sure:

• all nodes are running the same Skulk build
• for an explicit store deployment, you know which machine should be the store host and its chosen store_path is mounted, writable, and large enough
• for a default fresh cluster, the elected master's normal user-home filesystem has enough space for the models you want to share

The store server uses port 12415 by default. This listener is deliberately outside the dynamic client-port ranges used by supported operating systems, so an unrelated outbound connection cannot claim it before Skulk starts.

Fresh installs

install.sh writes bootstrap store defaults (this host under ~/.skulk/model-store) when no config exists, so a single node works immediately. When several independently installed nodes form a cluster, the elected master advertises a routable store address through bootstrap state sync. Followers adopt that authoritative config, stop their temporary local store servers, and point both dashboard and worker traffic at the same store. If you need a specific machine or attached volume, configure the same explicit store_host on every node instead.

Recommended Setup: Dashboard First

This is the simplest path for most people.

1. Start Skulk on all nodes with uv run skulk. Fresh defaults already converge on one store.
2. Open the dashboard on the node you want to use for administration.
3. Go to Settings.
4. To override the elected default, enable the store host toggle for the machine that should own the canonical store.
5. Choose its store path.
6. Save the config.
7. Restart Skulk on all nodes if the dashboard tells you a restart is required.

After that, use the dashboard or API normally. When models are available in the store, worker nodes stage from the store host instead of downloading independently.

Manual Setup with skulk.yaml

If you prefer to configure the model store manually, put the same skulk.yaml file on each node.

Minimal example:

model_store:
  enabled: true
  store_host: mac-studio-1
  store_path: /Volumes/ModelStore/models

For most users:

• store_host should be the hostname of the store machine
• store_path should be an absolute path on that host

Example Full Configuration

model_store:
  enabled: true
  store_host: mac-studio-1
  store_port: 12415
  store_path: /Volumes/ModelStore/models

  download:
    allow_hf_fallback: true

  staging:
    enabled: true
    node_cache_path: ~/.skulk/staging
    # Keep the newest ~40 GiB of idle staged copies warm and evict older ones
    # on deactivation/startup. Warm and bounded. Set false to keep everything
    # (unbounded) and reclaim disk only via POST /store/purge-staging.
    cleanup_on_deactivate: true
    staging_keep_recent_gb: 40

  reconciliation:
    enabled: true
    inventory_only: false
    interval_seconds: 300

  node_overrides:
    mac-studio-1:
      # The store host loads directly from the store path, so it makes no
      # second copy and the recency budget is skipped here regardless.
      staging:
        node_cache_path: /Volumes/ModelStore/models

How to Think About It

There are two important paths:

• store_path: the shared source of truth on the store host
• node_cache_path: the local staging area where a node prepares files before loading them

For worker nodes, node_cache_path is usually a fast local path such as ~/.skulk/staging.

For the store host, you often point node_cache_path at the same directory as store_path so the store host can load directly from the shared volume without making another copy.

Staging on the store host uses hardlinks

When the store host stages into a separate local staging directory (rather than pointing node_cache_path at the store), it hardlinks store files into place instead of copying them, falling back to a full copy only on filesystems that cannot link (a cross-device staging path, or some network mounts). This is safe because store files are immutable once registered and staged files are never modified in place.

The disk-usage consequence: when store and staging share a filesystem, a staged copy on the store host costs no additional disk; without the hardlink, a 26 GB GGUF would need 52 GB to stage. Note that naive per-directory size tools (du on each directory separately) double-count hardlinked files, so the store host's staging directory can look larger than the disk space it actually consumes. Worker nodes stage over HTTP from the store host, so their staged copies are always real bytes on their own disk.

Staging Cache and Disk Management

When a worker needs a model it does not host, it copies that model's files from the store host into its local staging directory (node_cache_path, default ~/.skulk/staging) and loads from there. Staged copies are independent per node: the store host keeps the canonical copy, and each worker keeps its own staged copy of whatever it has run. Left unmanaged, staging grows without bound, so Skulk combines a warm-cache recency policy with a pre-download free-space safety check and an explicit delete path.

What counts as "in use"

A staged copy is in use whenever a live runner depends on it, including companion repositories that no instance names directly (a speculative-decoding draft model, an assistant model, or separate vision weights). In-use copies are never evicted automatically.

The recency budget

Idle (not-in-use) staged copies are kept newest-first up to staging_keep_recent_gb (default 40 GiB); anything beyond that budget is deleted. This warm-cache check runs at two specific moments, and only when cleanup_on_deactivate is true:

• when a model instance is shut down, and
• at node startup, which reconciles copies orphaned by a crash or kill.

cleanup_on_deactivate is the on/off switch for that check:

Setting	Behavior
true (default, recommended)	Keep the newest ~40 GiB of idle copies warm; delete older idle copies when an instance shuts down and at node startup. The in-use set is always kept and does not count against the budget. Warm and bounded.
false	Skip lifecycle cleanup. Copies stay warm until reclaimed manually or a future download needs their space.

Set staging_keep_recent_gb to 0 for strict evict-on-deactivate (keep only what is in use). Raise it on nodes with large disks to keep more models warm.

The in-use set rides on top of the budget rather than inside it: a node always keeps everything its live runners need, plus up to 40 GiB of the most recently used idle copies.

Before any store probe or copy, staging checks the installed identity and local manifest. An exact complete cache is used immediately, including while air-gapped. A stale generation is replaced atomically from the central store, and missing required companions disable the fast path.

Pre-download capacity safety

For each base or companion repository, the store client resolves the exact registered artifact set and computes only the physical bytes the staging transaction will add. Manifest files and partial files already present reduce the HTTP allocation; same-filesystem local-store hardlinks add zero file data. Capacity admission and transfer are serialized per node so concurrent launches cannot spend the same free bytes. The worker retains 10 GiB of operating-system headroom and, if necessary, evicts idle copies oldest-first even when they are inside the 40 GiB warm-cache budget.

This safety pass protects live runners, every active base-plus-companion transaction, the incoming partial model, and its companion repositories. It also applies when cleanup_on_deactivate is false: that setting disables lifecycle cleanup, not the guard that prevents a new transfer from filling the host filesystem. If removing every idle staged model still cannot meet the target, Skulk reports DownloadFailed before writing more model bytes.

If the configured store is unreachable and Hugging Face fallback is enabled, the staging check does not run against the wrong filesystem. The direct downloader resolves the exact filtered artifact set, serializes admission with transfer, and applies the same reserve to the actual model-cache filesystem.

The store host is a special case. When a node points node_cache_path at the same directory as store_path (so it loads directly from the store without a second copy), the recency budget is skipped on that node whatever the toggle says. The store's canonical copies are never auto-evicted by either the lifecycle or capacity path. New canonical downloads are themselves serialized and admitted from their exact selected manifest, including resumable bytes, and fail before transfer if they cannot preserve the same 10 GiB reserve.

Deleting a model from the store

DELETE /store/models/{model_id} removes the canonical copy from the store host and evicts that model's staged copy from every node in the cluster at once. The eviction is a broadcast through the cluster's control plane: the delete issues a command, the master turns it into a fleet-wide event, and every worker that applies it removes its staged copy on disk while the model's download records are dropped from cluster state. This path is unconditional: it ignores both cleanup_on_deactivate and the recency budget, because once the canonical copy is gone the staged copies are orphans. Use it to remove a model everywhere and reclaim its disk fleet-wide in one call.

Reclaiming disk manually

POST /store/purge-staging clears staged copies on every node without touching the store's canonical copies. With no body it purges the whole staging cache; scoped to a model_id it purges just that model. Use it when cleanup_on_deactivate is false, or when you want space back immediately rather than waiting for the next deactivation.

Important Fields

model_store.enabled

Turns the model store on or off without deleting the config file.

model_store.store_host

The hostname or node ID of the store host.

For most users, hostname is the easiest and most reliable choice.

model_store.store_port

HTTP port used for store transfers.

Default: 12415

model_store.store_path

Absolute path on the store host where shared models live.

model_store.download.allow_hf_fallback

Controls what happens if a requested model is not already in the store, or the store cannot be reached at all.

Value	Behavior
true	Fall back to Hugging Face download when needed
false	Fail instead of downloading from Hugging Face

Use false if you want stricter offline or air-gapped behavior.

The fallback covers two distinct situations, and the node logs which one it is in:

• Model not in the store (the store answered): the node asks the store host to download it from Hugging Face, then stages from the store. The store stays the single source of truth for model files.
• Store unreachable (the store could not be reached, whether it never answered or dropped off mid-transfer): the node downloads directly from Hugging Face, preserving the card's pinned source revision. This is the expected shape for a remote fabric member whose route to the store does not exist while its public-internet path works; on a node that should be able to reach the store, treat the logged warning as a network problem to fix. With the fallback disabled, an unreachable store fails with an error naming unreachability rather than claiming the model is missing.

model_store.staging.node_cache_path

Where a node stages files before loading them.

model_store.staging.cleanup_on_deactivate

Controls automatic eviction of idle staged copies. Default true (recommended): when a model is shut down, or at node startup, idle staged copies beyond the staging_keep_recent_gb budget are deleted, keeping the cache warm but bounded. Set to false to skip those lifecycle passes. Pre-download capacity safety still reclaims idle copies when required to fit a new model without filling the filesystem. See Staging Cache and Disk Management.

model_store.staging.staging_keep_recent_gb

Recency budget in GiB for idle staged copies (default 40). Eviction keeps the newest idle copies up to this size and deletes the rest; 0 evicts everything not in use, and larger values keep more models warm on big-disk nodes. Applies to lifecycle cleanup when cleanup_on_deactivate is true; the pre-download capacity guard may override this grace budget.

Typical Flow

First time a model is needed

If the model is not already in the store and fallback is enabled:

1. Skulk requests the model.
2. The store-aware download path checks the store.
3. If the model is missing, Skulk falls back to Hugging Face.
4. The model lands in the appropriate local or store-managed path.

Later requests

Once the model exists in the store:

1. worker nodes ask the store host for the needed files
2. files are staged locally
3. inference loads from the staged path

Useful Store Endpoints

These are exposed through the main Skulk API:

• GET /store/health
• GET /store/registry
• GET /store/downloads
• GET /store/storage
• GET /store/reconciliation
• POST /store/reconciliation/rescan
• POST /store/models/{model_id}/download
• DELETE /store/models/{model_id}/download
• GET /store/models/{model_id}/download/status
• DELETE /store/models/{model_id}
• POST /store/purge-staging
• POST /store/models/{model_id}/optimize

The dashboard's Store Registry view combines these registry entries with model metadata so it can show capability-derived tags for downloaded models. Today that includes vision, thinking, embedding, tensor, and optiq when the underlying model card exposes enough metadata for Skulk to derive them.

DELETE /store/models/{model_id}/download stops pending or active canonical store transfer work without deleting its partial files. A later POST /store/models/{model_id}/download resumes from those partials. Repeating the cancellation is safe; a request for a model with no cancellable transfer returns 409.

POST /store/purge-staging recursively removes each selected node-cache artifact directory, including its adjacent .skulk/installed-card.json, revision marker, and last-use marker. It does not remove the canonical store generation. DELETE /store/models/{model_id} removes the canonical generation and broadcasts the corresponding staged-cache eviction across the fleet.

Common meanings:

• 503 Store not configured: the cluster is not configured to use a model store
• 503 Store unreachable: the store is configured, but the API cannot reach it
• 404: the model or job does not exist
• 409: a conflicting operation is already in progress

Troubleshooting

The store host seems unreachable

Check:

• that the store host is running
• that store_host matches the real hostname
• that port 12415 is reachable on your LAN

Useful check:

curl http://STORE_HOST:12415/health

The model is on disk but does not appear in the store registry

Check:

• that the model is in the configured store_path
• that the registry knows about it
• that the dashboard Store Registry view shows it

Useful check:

curl http://localhost:52415/store/registry

Nodes still download from Hugging Face

Check:

• whether the model is already present in the store
• whether allow_hf_fallback is still true
• whether the store host is reachable from worker nodes

A multimodal model is in the store but the UI does not show vision support

Check:

• that the model card includes the vision capability
• that the dashboard is running a current Skulk build
• that GET /v1/models returns a vision tag for that model

Remember that store registration only tracks artifacts and metadata. Actual image understanding still depends on launching the model and sending a multimodal request through the chat APIs.

Placements are slow even though the model is already in the store

A staged copy that falls outside the recency budget had to be re-copied from the store host before loading. With cleanup_on_deactivate: true (the default), the newest ~40 GiB of idle copies stay warm, but a model larger than the budget, or one pushed out by other recently used models, is evicted and re-staged on its next placement. Raise staging_keep_recent_gb to keep more models warm, or set it large enough to hold the models you cycle between. Setting cleanup_on_deactivate: false keeps every copy warm until a future launch needs that space; the pre-download safety pass can still evict idle copies to protect the filesystem (reclaim disk immediately with POST /store/purge-staging).

Staged files are not being cleaned up

With cleanup_on_deactivate: false, lifecycle cleanup does not run, so staging keeps growing until a new transfer needs the space. Either leave it at the true default (which keeps the newest ~40 GiB warm and evicts the rest on deactivation/startup) or reclaim space on demand with POST /store/purge-staging. The pre-download capacity guard still protects the filesystem. Note that the store host never auto-evicts its own canonical copies when node_cache_path equals store_path.

Good Defaults for Most Clusters

• use the dashboard to manage the store config
• choose one machine with the most storage as the store host
• keep allow_hf_fallback: true while you are getting started
• use a fast local staging path on worker nodes
• point the store host's node_cache_path at the store itself

Related Docs

• README
• API guide
• Architecture overview
• skulk.yaml example