Skip to main content

ClusterNodeDiagnostics

One node result inside a cluster diagnostics response.

nodeIdNodeid (string)required

Node ID for this cluster diagnostics result.

url object

Peer API URL used to collect diagnostics, if remote.

anyOf
string
okOk (boolean)required

Whether diagnostics were collected successfully.

versionStatusVersionstatus (string)

Whether this node reports the same known Skulk version and build as the API collecting the cluster response.

Possible values: [current, version_mismatch, unknown]

Default value: unknown
diagnostics object

Collected node diagnostics when ok is true.

anyOf
generatedAtGeneratedat (string)required

UTC timestamp when this bundle was built.

runtime objectrequired

Runtime identity and config.

nodeIdNodeid (string)required

Local node ID.

hostnameHostname (string)required

Local hostname.

friendlyName object

Friendly node name from gathered identity data.

anyOf
string
isMasterIsmaster (boolean)required

Whether this node is the current master.

masterNodeId object

Current master node ID, when known.

anyOf
string
cwdCwd (string)required

Current working directory of the API process.

configPathConfigpath (string)required

Config path resolved by this API process.

configFileExistsConfigfileexists (boolean)required

Whether the resolved config path exists from this process cwd.

skulkVersionSkulkversion (string)required

Installed Skulk package version.

skulkCommitSkulkcommit (string)required

Git commit reported by node identity.

libp2PNamespace object

Configured libp2p namespace environment value, if set.

anyOf
string
pythonUnbufferedPythonunbuffered (boolean)required

Whether PYTHONUNBUFFERED is enabled for this process.

tracingEnabledTracingenabled (boolean)required

Current cluster runtime tracing state as seen by this API node.

structuredLoggingConfiguredStructuredloggingconfigured (boolean)required

Whether config enables centralized structured logging.

loggingIngestUrl object

Configured centralized logging ingest URL, when present.

anyOf
string
identity object

Last gathered node identity data.

anyOf
modelIdModelid (string)
Default value: Unknown
chipIdChipid (string)
Default value: Unknown
friendlyNameFriendlyname (string)
Default value: Unknown
osVersionOsversion (string)
Default value: Unknown
osBuildVersionOsbuildversion (string)
Default value: Unknown
skulkVersionSkulkversion (string)
Default value: Unknown
skulkCommitSkulkcommit (string)
Default value: Unknown
resources objectrequired

Resource readings.

gatheredMemory object

Last event-sourced memory reading for this node.

anyOf
ramTotal objectrequired
inBytesInbytes (integer)
Default value: 0
ramAvailable objectrequired
inBytesInbytes (integer)
Default value: 0
swapTotal objectrequired
inBytesInbytes (integer)
Default value: 0
swapAvailable objectrequired
inBytesInbytes (integer)
Default value: 0
currentMemory object

Live memory reading from the API process.

anyOf
ramTotal objectrequired
inBytesInbytes (integer)
Default value: 0
ramAvailable objectrequired
inBytesInbytes (integer)
Default value: 0
swapTotal objectrequired
inBytesInbytes (integer)
Default value: 0
swapAvailable objectrequired
inBytesInbytes (integer)
Default value: 0
currentWired object

Live OS-level wired (unpageable) memory in use (macOS only). Read locally on this endpoint — deliberately NOT on the gossiped MemoryUsage, whose schema rides extra=forbid events — to detect leaked wired memory after an abnormal Metal termination (#239).

anyOf
inBytesInbytes (integer)
Default value: 0
disk object

Last event-sourced disk reading for this node.

anyOf
total objectrequired
inBytesInbytes (integer)
Default value: 0
available objectrequired
inBytesInbytes (integer)
Default value: 0
system object

Last event-sourced system performance reading.

anyOf
gpuUsageGpuusage (number)
Default value: 0
tempTemp (number)
Default value: 0
sysPowerSyspower (number)
Default value: 0
pcpuUsagePcpuusage (number)
Default value: 0
ecpuUsageEcpuusage (number)
Default value: 0
accelerator object
anyOf
vendorVendor (string)

Possible values: [apple, amd, nvidia, intel, cpu, unknown]

Default value: unknown
nameName (string)
Default value: Unknown
utilizationRatio object
anyOf
number
vramTotalBytes object
anyOf
integer
vramUsedBytes object
anyOf
integer
gttTotalBytes object

GPU-mappable host (GTT) memory, for unified-memory APUs (e.g. AMD Strix Halo). On such a node the GPU addresses system RAM beyond the BIOS VRAM carve-out through GTT, so the usable GPU pool is far larger than vram_total_bytes (placement uses this to admit big models on a UMA node). None on discrete GPUs / collectors that do not report it.

anyOf
integer
powerWatts object
anyOf
number
temperatureCelsius object
anyOf
number
clockMhz object
anyOf
integer
computeCapability object

Discrete-GPU compute capability as "<major>.<minor>" (NVIDIA SM level), e.g. "8.0" (A100 Ampere), "9.0" (H100 Hopper), "10.0" (B100/B200 Blackwell), "12.0" (RTX 50 Blackwell). The engine/quant/placement decision keys on this, not on vendor: the same model+engine performs oppositely across generations (a benchmark showed vLLM's MXFP4 path losing single-stream on Ampere, which has no native FP4, but winning on Blackwell, which does). None on collectors that do not report it (AMD sysfs, Apple).

anyOf
string
nativeFp4 object

Whether the GPU accelerates FP4 natively (Blackwell sm100+, i.e. SM level (major, minor) >= (10, 0)). Derived at the collector boundary from the parsed compute capability (a numeric tuple compare, not a string compare). None when unmeasured.

anyOf
boolean
nativeFp8 object

Whether the GPU accelerates FP8 natively (Ada sm89 / Hopper sm90 and later). Derived at the collector boundary from the compute capability. None when unmeasured.

anyOf
boolean
network object

Last event-sourced network interface reading.

anyOf
interfaces object[]
  • Array [
  • nameName (string)required
    ipAddressIpaddress (string)required
    interfaceTypeInterfacetype (string)

    Possible values: [wifi, ethernet, maybe_ethernet, thunderbolt, unknown]

    Default value: unknown
  • ]
  • processes object[]

    Relevant local OS processes.

  • Array [
  • pidPid (integer)required

    Operating-system process ID.

    parentPid object

    Operating-system parent process ID, when visible.

    anyOf
    integer
    roleRole (string)required

    Best-effort Skulk role inferred from process lineage and command line.

    Possible values: [skulk, runner, vector, python, other]

    commandCommand (string)required

    Joined process command line.

    status object

    Operating-system process status such as running or sleeping.

    anyOf
    string
    cpuPercent object

    Recent CPU percentage reported by psutil.

    anyOf
    number
    memoryPercent object

    Percent of physical memory used by this process.

    anyOf
    number
    rss object

    Resident set size for this process, when available.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    elapsedSeconds object

    Seconds since process creation, when available.

    anyOf
    number
    isChildOfSkulkIschildofskulk (boolean)

    Whether this process is in the current Skulk API process tree.

    Default value: false
  • ]
  • supervisorRunners object[]

    Live local runner-supervisor diagnostics.

  • Array [
  • runnerIdRunnerid (string)required

    Runner ID.

    instanceIdInstanceid (string)required

    Instance ID.

    nodeIdNodeid (string)required

    Node ID that owns this runner.

    modelIdModelid (string)required

    Model assigned to this runner.

    deviceRankDevicerank (integer)required

    Distributed device rank.

    worldSizeWorldsize (integer)required

    Distributed world size.

    startLayerStartlayer (integer)required

    Inclusive first model layer on this shard.

    endLayerEndlayer (integer)required

    Exclusive final model layer on this shard.

    nLayersNlayers (integer)required

    Total number of model layers.

    pid object

    Runner subprocess PID, when started.

    anyOf
    integer
    processAliveProcessalive (boolean)required

    Whether the runner subprocess is alive.

    exitCode object

    Runner subprocess exit code, when exited.

    anyOf
    integer
    statusKindStatuskind (string)required

    Current runner status variant.

    statusSinceStatussince (string)required

    UTC timestamp for the current status.

    secondsInStatusSecondsinstatus (number)required

    Wall-clock seconds spent in the current runner status.

    phasePhase (string)required

    Last runner phase reported.

    Possible values: [created, idle, connect_group, load_model, warmup, task_submission, task_agreement, prompt_build, vision_preprocess, kv_cache_lookup, prefill_barrier, prefill_pipeline, prefill_stream, decode_barrier, decode_wait_first_token, decode_stream, parser, cancel_requested, cancel_observed, completion, error, shutdown_cleanup]

    phaseStartedAtPhasestartedat (string)required

    UTC timestamp when the current phase started.

    secondsInPhaseSecondsinphase (number)required

    Wall-clock seconds spent in the current phase.

    lastProgressAt object

    UTC timestamp for the last flight-recorder update.

    anyOf
    string
    activeTaskId object

    Task ID associated with the current phase, when known.

    anyOf
    string
    activeCommandId object

    Command ID associated with the current phase, when known.

    anyOf
    string
    phaseDetail object

    Compact human-readable detail for the current phase.

    anyOf
    string
    lastMlxMemory object

    Most recent MLX memory snapshot reported by the runner.

    anyOf
    generatedAtGeneratedat (string)required

    UTC timestamp when the snapshot was taken.

    active object

    Currently active MLX memory, when the runtime exposes it.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    cache object

    MLX cache memory, when the runtime exposes it.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    peak object

    Peak MLX memory since the last reset, when available.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    wiredLimit object

    Configured MLX wired memory limit when known. Current MLX releases do not expose a getter on all platforms, so this may be null.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    sourceSource (string)required

    Runtime module that supplied the measurement, such as mlx.core.

    flightRecorder object[]

    Last 128 local-only runner diagnostic events.

  • Array [
  • atAt (string)required

    UTC timestamp when the runner emitted the update.

    phasePhase (string)required

    Runner phase at this entry.

    Possible values: [created, idle, connect_group, load_model, warmup, task_submission, task_agreement, prompt_build, vision_preprocess, kv_cache_lookup, prefill_barrier, prefill_pipeline, prefill_stream, decode_barrier, decode_wait_first_token, decode_stream, parser, cancel_requested, cancel_observed, completion, error, shutdown_cleanup]

    eventEvent (string)required

    Short event name within the phase.

    detail object

    Compact human-readable detail for diagnostics.

    anyOf
    string
    attrs object

    Structured low-cardinality diagnostic attributes.

    property name* object
    anyOf
    string
    context objectrequired

    Stable runner identity fields for this entry.

    nodeIdNodeid (string)required

    Node ID that owns this runner.

    runnerIdRunnerid (string)required

    Runner ID.

    pid object

    Runner subprocess PID.

    anyOf
    integer
    instanceIdInstanceid (string)required

    Instance ID.

    modelIdModelid (string)required

    Model assigned to this runner.

    rankRank (integer)required

    Distributed rank for this runner.

    worldSizeWorldsize (integer)required

    Distributed world size.

    startLayerStartlayer (integer)required

    Inclusive first layer on this shard.

    endLayerEndlayer (integer)required

    Exclusive final layer on this shard.

    nLayersNlayers (integer)required

    Total model layers.

    taskId object

    Task ID associated with the entry, when known.

    anyOf
    string
    commandId object

    Command ID associated with the entry, when known.

    anyOf
    string
    mlxMemory object

    MLX memory snapshot captured with this entry, when present.

    anyOf
    generatedAtGeneratedat (string)required

    UTC timestamp when the snapshot was taken.

    active object

    Currently active MLX memory, when the runtime exposes it.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    cache object

    MLX cache memory, when the runtime exposes it.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    peak object

    Peak MLX memory since the last reset, when available.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    wiredLimit object

    Configured MLX wired memory limit when known. Current MLX releases do not expose a getter on all platforms, so this may be null.

    anyOf
    inBytesInbytes (integer)
    Default value: 0
    sourceSource (string)required

    Runtime module that supplied the measurement, such as mlx.core.

  • ]
  • pendingTaskIdsstring[]

    Tasks sent to the supervisor but not acknowledged by the runner.

    inProgressTasks object[]

    Tasks currently known as in progress by the supervisor.

  • Array [
  • taskIdTaskid (string)required

    Skulk task ID.

    taskKindTaskkind (string)required

    Concrete task model name.

    taskStatusTaskstatus (string)required

    Current event-sourced task status.

    instanceIdInstanceid (string)required

    Instance associated with the task.

    commandId object

    External command ID for user-facing inference tasks.

    anyOf
    string
    runnerId object

    Runner assigned to the task, if known.

    anyOf
    string
    modelId object

    Model associated with the task, if known.

    anyOf
    string
  • ]
  • completedTaskCountCompletedtaskcount (integer)required

    Number of tasks completed by this supervisor.

    cancelledTaskIdsstring[]

    Task IDs cancelled through this supervisor.

    lastTaskSentAt object

    UTC timestamp for the last task submitted to the runner.

    anyOf
    string
    lastEventReceivedAt object

    UTC timestamp for the last event received from the runner.

    anyOf
    string
    lastEventType object

    Class name of the last event received from the runner.

    anyOf
    string
    milestones object[]

    Recent lifecycle milestones retained by the supervisor.

  • Array [
  • atAt (string)required

    UTC timestamp when the milestone was recorded.

    nameName (string)required

    Short milestone name.

    detail object

    Optional compact detail for the milestone.

    anyOf
    string
  • ]
  • ]
  • placements object[]

    Event-sourced placement analysis for current instances.

  • Array [
  • instanceIdInstanceid (string)required

    Instance ID.

    modelIdModelid (string)required

    Placed model ID.

    masterNodeId object

    Current master node ID, when known.

    anyOf
    string
    masterIsPlacementNodeMasterisplacementnode (boolean)required

    Whether the current master is part of this model placement.

    localNodeIsPlacementNodeLocalnodeisplacementnode (boolean)required

    Whether the API node is part of this model placement.

    placementNodeIdsstring[]

    Node IDs participating in the placement.

    runners object[]

    Per-runner placement details.

  • Array [
  • runnerIdRunnerid (string)required

    Runner ID.

    nodeIdNodeid (string)required

    Node ID assigned to this runner.

    friendlyName object

    Friendly node name, when known.

    anyOf
    string
    statusKind object

    Current event-sourced runner status variant.

    anyOf
    string
    deviceRankDevicerank (integer)required

    Distributed device rank.

    worldSizeWorldsize (integer)required

    Distributed world size.

    startLayerStartlayer (integer)required

    Inclusive first model layer on this shard.

    endLayerEndlayer (integer)required

    Exclusive final model layer on this shard.

    nLayersNlayers (integer)required

    Total number of model layers.

    isLocalIslocal (boolean)required

    Whether this assignment is on the API node.

    isMasterIsmaster (boolean)required

    Whether this assignment is on the master node.

    tasks object[]

    Event-sourced tasks associated with this runner assignment.

  • Array [
  • taskIdTaskid (string)required

    Skulk task ID.

    taskKindTaskkind (string)required

    Concrete task model name.

    taskStatusTaskstatus (string)required

    Current event-sourced task status.

    instanceIdInstanceid (string)required

    Instance associated with the task.

    commandId object

    External command ID for user-facing inference tasks.

    anyOf
    string
    runnerId object

    Runner assigned to the task, if known.

    anyOf
    string
    modelId object

    Model associated with the task, if known.

    anyOf
    string
  • ]
  • ]
  • warningsstring[]

    Heuristic warnings that may help explain a stuck placement.

  • ]
  • dataPlane objectrequired

    Local DATA stream lifecycle, ordering, and timing metrics.

    transportTransport (string)required

    Configured DATA transport on this API node.

    Possible values: [disabled, gossipsub, zenoh]

    reorderBufferEnabledReorderbufferenabled (boolean)required

    Whether this API reorders DATA frames before dispatch.

    activeStreamsActivestreams (integer)required

    Streams accepted without a terminal frame yet.

    framesReceivedFramesreceived (integer)required

    All DATA frames observed before deduplication.

    framesDispatchedFramesdispatched (integer)required

    Ordered, unique DATA frames accepted for live commands.

    startedFramesStartedframes (integer)required

    Accepted started lifecycle frames.

    chunkFramesChunkframes (integer)required

    Accepted non-terminal payload frames.

    completedFramesCompletedframes (integer)required

    Accepted completed terminal frames.

    failedFramesFailedframes (integer)required

    Accepted failed terminal frames.

    cancelledFramesCancelledframes (integer)required

    Accepted cancelled terminal frames.

    duplicateFramesDuplicateframes (integer)required

    Frames dropped because their sequence was already observed.

    outOfOrderFramesOutoforderframes (integer)required

    Frames received above the next expected sequence.

    skippedSequencesSkippedsequences (integer)required

    Missing sequence numbers skipped after a bounded reorder wait.

    lateFramesLateframes (integer)required

    Frames dropped because no live command queue remained.

    missingStartedStreamsMissingstartedstreams (integer)required

    Streams whose first accepted frame was not started.

    missingTerminalStreamsMissingterminalstreams (integer)required

    Streams finalized locally without an accepted terminal frame.

    idleTimeoutsIdletimeouts (integer)required

    API stream receives that exceeded the DATA idle deadline.

    transportFailuresTransportfailures (integer)required

    Streams failed locally after a DATA ordering or delivery gap.

    firstByteSamplesFirstbytesamples (integer)required

    Streams with a measured started-to-first-payload interval.

    firstByteSecondsLast object

    Most recently measured started-to-first-payload latency.

    anyOf
    number
    firstByteSecondsAverage object

    Average started-to-first-payload latency for this process.

    anyOf
    number
    firstByteSecondsMax object

    Maximum started-to-first-payload latency for this process.

    anyOf
    number
    streamSpanSamplesStreamspansamples (integer)required

    Terminal streams with a measured first-payload-to-terminal span.

    streamSpanSecondsLast object

    Most recently measured first-payload-to-terminal span.

    anyOf
    number
    streamSpanSecondsAverage object

    Average first-payload-to-terminal span for this process.

    anyOf
    number
    streamSpanSecondsMax object

    Maximum first-payload-to-terminal span for this process.

    anyOf
    number
    egress objectrequired

    Router-side DATA queue, publish, and isolation metrics.

    activeStreamQueuesActivestreamqueues (integer)required

    Independent remote command queues currently active.

    queueDepthQueuedepth (integer)required

    Frames currently queued across remote command streams.

    maxQueueDepthMaxqueuedepth (integer)required

    Highest aggregate remote DATA queue depth observed.

    localShortCircuitsLocalshortcircuits (integer)required

    Frames delivered locally without remote Zenoh egress.

    remoteFramesEnqueuedRemoteframesenqueued (integer)required

    Remote frames accepted into bounded command queues.

    remoteFramesPublishedRemoteframespublished (integer)required

    Remote frames successfully published over Zenoh.

    remoteFramesDroppedRemoteframesdropped (integer)required

    Remote frames dropped by queue or stream admission bounds.

    remotePublishFailuresRemotepublishfailures (integer)required

    Remote publishes that raised or exceeded their deadline.

    remoteBytesPublishedRemotebytespublished (integer)required

    Serialized DATA bytes successfully published over Zenoh.

    inboundPayloadQueueDepthInboundpayloadqueuedepth (integer)

    Payload frames waiting in an isolated network-ingress lane.

    Default value: 0
    inboundPayloadQueueCapacityInboundpayloadqueuecapacity (integer)

    Hard capacity of the isolated payload ingress lane.

    Default value: 0
    inboundTerminalQueueDepthInboundterminalqueuedepth (integer)

    Terminal frames waiting in an isolated network-ingress lane.

    Default value: 0
    inboundTerminalQueueCapacityInboundterminalqueuecapacity (integer)

    Hard capacity of the isolated terminal ingress lane.

    Default value: 0
    inboundFramesDroppedInboundframesdropped (integer)

    Network-ingress frames rejected by isolated lane bounds.

    Default value: 0
    idleStreamReclaimsIdlestreamreclaims (integer)

    Remote command queues reclaimed after their idle lease expired.

    Default value: 0
    enqueueLatencySamplesEnqueuelatencysamples (integer)required

    Remote frames with measured TopicRouter enqueue latency.

    enqueueLatencySecondsAverage object

    Average TopicRouter-to-egress enqueue latency.

    anyOf
    number
    enqueueLatencySecondsMax object

    Maximum TopicRouter-to-egress enqueue latency.

    anyOf
    number
    publishLatencySamplesPublishlatencysamples (integer)required

    Remote frames with measured Zenoh publish latency.

    publishLatencySecondsAverage object

    Average successful or failed Zenoh publish latency.

    anyOf
    number
    publishLatencySecondsMax object

    Maximum successful or failed Zenoh publish latency.

    anyOf
    number
    owners object

    Per-owner pressure counters keyed by opaque node id.

    property name* DataPlaneOwnerDiagnostics

    Bounded egress pressure metrics for one remote DATA owner.

    queueDepthQueuedepth (integer)required

    Frames currently queued for this owner.

    activeStreamsActivestreams (integer)required

    Independent command egress queues active for this owner.

    maxQueueDepthMaxqueuedepth (integer)required

    Highest aggregate queue depth observed for this owner.

    framesEnqueuedFramesenqueued (integer)required

    Frames accepted into this owner's command queues.

    framesPublishedFramespublished (integer)required

    Frames successfully published for this owner.

    framesDroppedFramesdropped (integer)required

    Frames rejected by this owner's bounded queues.

    publishFailuresPublishfailures (integer)required

    Publish exceptions or deadline expirations for this owner.

    idleStreamReclaimsIdlestreamreclaims (integer)

    Command queues reclaimed after their egress idle lease expired.

    Default value: 0
    visionMediaEgress object

    Isolated router egress pressure for inbound VLM and image-edit media.

    activeStreamQueuesActivestreamqueues (integer)required

    Independent remote command queues currently active.

    queueDepthQueuedepth (integer)required

    Frames currently queued across remote command streams.

    maxQueueDepthMaxqueuedepth (integer)required

    Highest aggregate remote DATA queue depth observed.

    localShortCircuitsLocalshortcircuits (integer)required

    Frames delivered locally without remote Zenoh egress.

    remoteFramesEnqueuedRemoteframesenqueued (integer)required

    Remote frames accepted into bounded command queues.

    remoteFramesPublishedRemoteframespublished (integer)required

    Remote frames successfully published over Zenoh.

    remoteFramesDroppedRemoteframesdropped (integer)required

    Remote frames dropped by queue or stream admission bounds.

    remotePublishFailuresRemotepublishfailures (integer)required

    Remote publishes that raised or exceeded their deadline.

    remoteBytesPublishedRemotebytespublished (integer)required

    Serialized DATA bytes successfully published over Zenoh.

    inboundPayloadQueueDepthInboundpayloadqueuedepth (integer)

    Payload frames waiting in an isolated network-ingress lane.

    Default value: 0
    inboundPayloadQueueCapacityInboundpayloadqueuecapacity (integer)

    Hard capacity of the isolated payload ingress lane.

    Default value: 0
    inboundTerminalQueueDepthInboundterminalqueuedepth (integer)

    Terminal frames waiting in an isolated network-ingress lane.

    Default value: 0
    inboundTerminalQueueCapacityInboundterminalqueuecapacity (integer)

    Hard capacity of the isolated terminal ingress lane.

    Default value: 0
    inboundFramesDroppedInboundframesdropped (integer)

    Network-ingress frames rejected by isolated lane bounds.

    Default value: 0
    idleStreamReclaimsIdlestreamreclaims (integer)

    Remote command queues reclaimed after their idle lease expired.

    Default value: 0
    enqueueLatencySamplesEnqueuelatencysamples (integer)required

    Remote frames with measured TopicRouter enqueue latency.

    enqueueLatencySecondsAverage object

    Average TopicRouter-to-egress enqueue latency.

    anyOf
    number
    enqueueLatencySecondsMax object

    Maximum TopicRouter-to-egress enqueue latency.

    anyOf
    number
    publishLatencySamplesPublishlatencysamples (integer)required

    Remote frames with measured Zenoh publish latency.

    publishLatencySecondsAverage object

    Average successful or failed Zenoh publish latency.

    anyOf
    number
    publishLatencySecondsMax object

    Maximum successful or failed Zenoh publish latency.

    anyOf
    number
    owners object

    Per-owner pressure counters keyed by opaque node id.

    property name* DataPlaneOwnerDiagnostics

    Bounded egress pressure metrics for one remote DATA owner.

    queueDepthQueuedepth (integer)required

    Frames currently queued for this owner.

    activeStreamsActivestreams (integer)required

    Independent command egress queues active for this owner.

    maxQueueDepthMaxqueuedepth (integer)required

    Highest aggregate queue depth observed for this owner.

    framesEnqueuedFramesenqueued (integer)required

    Frames accepted into this owner's command queues.

    framesPublishedFramespublished (integer)required

    Frames successfully published for this owner.

    framesDroppedFramesdropped (integer)required

    Frames rejected by this owner's bounded queues.

    publishFailuresPublishfailures (integer)required

    Publish exceptions or deadline expirations for this owner.

    idleStreamReclaimsIdlestreamreclaims (integer)

    Command queues reclaimed after their egress idle lease expired.

    Default value: 0
    visionMediaIngress object

    Bounded worker-side image upload occupancy and outcomes.

    pendingApiCommandsPendingapicommands (integer)required

    Image commands retained by the API while awaiting placement.

    pendingApiBytesPendingapibytes (integer)required

    Image bytes retained by the API while awaiting placement.

    activeApiCommandsActiveapicommands (integer)required

    Image transfers sent and still awaiting worker verification.

    activeApiBytesActiveapibytes (integer)required

    Source image bytes charged to active transfer admission.

    pendingWorkerAcknowledgementsPendingworkeracknowledgements (integer)required

    Selected worker verifications still owed to the API.

    activeStreamsActivestreams (integer)required

    Incomplete or verified image streams retained by this worker.

    pendingFramesPendingframes (integer)required

    Unverified image frames currently retained.

    retainedBytesRetainedbytes (integer)required

    Image bytes retained across pending and verified streams.

    verifiedStreamsVerifiedstreams (integer)required

    Verified streams waiting for runner dispatch.

    pendingFailuresPendingfailures (integer)required

    Bounded rejected-stream failures awaiting task correlation.

    completedStreamsCompletedstreams (integer)required

    Streams that passed integrity verification since process start.

    rejectedStreamsRejectedstreams (integer)required

    Streams rejected for bounds, metadata, or integrity failures.

    expiredStreamsExpiredstreams (integer)required

    Streams rejected after exceeding the ingress age limit.

    provider object

    Local provider admission, lifecycle, media, and pressure metrics.

    activeStreamsActivestreams (integer)required

    Currently admitted provider streams.

    maxActiveStreamsMaxactivestreams (integer)required

    Highest concurrent admitted stream count observed.

    admittedStreamsAdmittedstreams (integer)required

    Provider streams admitted for execution.

    rejectedStreamsRejectedstreams (integer)required

    Provider stream opens rejected after capability resolution.

    overloadedRejectionsOverloadedrejections (integer)required

    Rejected opens caused by the provider concurrency bound.

    inputQueueDepthInputqueuedepth (integer)required

    Caller input frames currently buffered before provider handling.

    maxInputQueueDepthMaxinputqueuedepth (integer)required

    Highest caller input queue depth observed.

    inputFramesInputframes (integer)required

    Caller-to-provider lifecycle frames delivered to the handler.

    inputMediaBytesInputmediabytes (integer)required

    Inline media bytes delivered from callers to the handler.

    outputFramesOutputframes (integer)required

    Provider-to-caller lifecycle frames accepted for DATA egress.

    outputMediaBytesOutputmediabytes (integer)required

    Inline media bytes accepted from the provider for DATA egress.

    completedStreamsCompletedstreams (integer)required

    Streams ending with completed.

    failedStreamsFailedstreams (integer)required

    Streams ending with failed.

    cancelledStreamsCancelledstreams (integer)required

    Streams ending with cancelled.

    missingTerminalStreamsMissingterminalstreams (integer)required

    Admitted streams removed without an observable terminal frame.

    cancellationRequestsCancellationrequests (integer)required

    Valid cancellation requests received for active streams.

    firstOutputSamplesFirstoutputsamples (integer)required

    Streams with a measured admission-to-first-output interval.

    firstOutputSecondsAverage object

    Average admission-to-first-output latency.

    anyOf
    number
    firstOutputSecondsMax object

    Maximum admission-to-first-output latency.

    anyOf
    number
    durationSamplesDurationsamples (integer)required

    Terminal streams with a measured admitted lifetime.

    durationSecondsAverage object

    Average admitted stream lifetime.

    anyOf
    number
    durationSecondsMax object

    Maximum admitted stream lifetime.

    anyOf
    number
    activeUnaryCallsActiveunarycalls (integer)required

    Generic unary provider calls currently executing.

    streamSlotsInUseStreamslotsinuse (integer)required

    Provider stream slots reserved by admission or execution.

    unaryConcurrencyLimitUnaryconcurrencylimit (integer)required

    Configured process-wide unary provider concurrency bound.

    streamConcurrencyLimitStreamconcurrencylimit (integer)required

    Configured process-wide streaming provider concurrency bound.

    capabilities object

    Metrics keyed by stable qualified capability id.

    property name* ProviderCapabilityDiagnostics

    Bounded lifecycle and media metrics for one served capability.

    activeStreamsActivestreams (integer)required

    Currently admitted provider streams.

    maxActiveStreamsMaxactivestreams (integer)required

    Highest concurrent admitted stream count observed.

    admittedStreamsAdmittedstreams (integer)required

    Provider streams admitted for execution.

    rejectedStreamsRejectedstreams (integer)required

    Provider stream opens rejected after capability resolution.

    overloadedRejectionsOverloadedrejections (integer)required

    Rejected opens caused by the provider concurrency bound.

    inputQueueDepthInputqueuedepth (integer)required

    Caller input frames currently buffered before provider handling.

    maxInputQueueDepthMaxinputqueuedepth (integer)required

    Highest caller input queue depth observed.

    inputFramesInputframes (integer)required

    Caller-to-provider lifecycle frames delivered to the handler.

    inputMediaBytesInputmediabytes (integer)required

    Inline media bytes delivered from callers to the handler.

    outputFramesOutputframes (integer)required

    Provider-to-caller lifecycle frames accepted for DATA egress.

    outputMediaBytesOutputmediabytes (integer)required

    Inline media bytes accepted from the provider for DATA egress.

    completedStreamsCompletedstreams (integer)required

    Streams ending with completed.

    failedStreamsFailedstreams (integer)required

    Streams ending with failed.

    cancelledStreamsCancelledstreams (integer)required

    Streams ending with cancelled.

    missingTerminalStreamsMissingterminalstreams (integer)required

    Admitted streams removed without an observable terminal frame.

    cancellationRequestsCancellationrequests (integer)required

    Valid cancellation requests received for active streams.

    firstOutputSamplesFirstoutputsamples (integer)required

    Streams with a measured admission-to-first-output interval.

    firstOutputSecondsAverage object

    Average admission-to-first-output latency.

    anyOf
    number
    firstOutputSecondsMax object

    Maximum admission-to-first-output latency.

    anyOf
    number
    durationSamplesDurationsamples (integer)required

    Terminal streams with a measured admitted lifetime.

    durationSecondsAverage object

    Average admitted stream lifetime.

    anyOf
    number
    durationSecondsMax object

    Maximum admitted stream lifetime.

    anyOf
    number
    warningsstring[]

    Top-level diagnostic warnings for this node.

    tailscale object

    This node's own Tailscale state. The probe is best-effort: a missing CLI, timeout, or unparsable output reads as running=false; None marks only an unexpected probe exception.

    anyOf
    runningRunning (boolean)required

    Whether tailscaled reports BackendState Running.

    selfIp object

    Tailscale IPv4 (100.x) when running.

    anyOf
    string
    hostname object

    Tailnet-registered hostname.

    anyOf
    string
    dnsName object

    MagicDNS name when available.

    anyOf
    string
    error object

    Collection error when ok is false.

    anyOf
    string
    ClusterNodeDiagnostics
    {
    "nodeId": "string",
    "url": "string",
    "ok": true,
    "versionStatus": "unknown",
    "diagnostics": {
    "generatedAt": "string",
    "runtime": {
    "nodeId": "string",
    "hostname": "string",
    "friendlyName": "string",
    "isMaster": true,
    "masterNodeId": "string",
    "cwd": "string",
    "configPath": "string",
    "configFileExists": true,
    "skulkVersion": "string",
    "skulkCommit": "string",
    "libp2PNamespace": "string",
    "pythonUnbuffered": true,
    "tracingEnabled": true,
    "structuredLoggingConfigured": true,
    "loggingIngestUrl": "string"
    },
    "identity": {
    "modelId": "Unknown",
    "chipId": "Unknown",
    "friendlyName": "Unknown",
    "osVersion": "Unknown",
    "osBuildVersion": "Unknown",
    "skulkVersion": "Unknown",
    "skulkCommit": "Unknown"
    },
    "resources": {
    "gatheredMemory": {
    "ramTotal": {
    "inBytes": 0
    },
    "ramAvailable": {
    "inBytes": 0
    },
    "swapTotal": {
    "inBytes": 0
    },
    "swapAvailable": {
    "inBytes": 0
    }
    },
    "currentMemory": {
    "ramTotal": {
    "inBytes": 0
    },
    "ramAvailable": {
    "inBytes": 0
    },
    "swapTotal": {
    "inBytes": 0
    },
    "swapAvailable": {
    "inBytes": 0
    }
    },
    "currentWired": {
    "inBytes": 0
    },
    "disk": {
    "total": {
    "inBytes": 0
    },
    "available": {
    "inBytes": 0
    }
    },
    "system": {
    "gpuUsage": 0,
    "temp": 0,
    "sysPower": 0,
    "pcpuUsage": 0,
    "ecpuUsage": 0,
    "accelerator": {
    "vendor": "unknown",
    "name": "Unknown",
    "utilizationRatio": 0,
    "vramTotalBytes": 0,
    "vramUsedBytes": 0,
    "gttTotalBytes": 0,
    "powerWatts": 0,
    "temperatureCelsius": 0,
    "clockMhz": 0,
    "computeCapability": "string",
    "nativeFp4": true,
    "nativeFp8": true
    }
    },
    "network": {
    "interfaces": [
    {
    "name": "string",
    "ipAddress": "string",
    "interfaceType": "unknown"
    }
    ]
    }
    },
    "processes": [
    {
    "pid": 0,
    "parentPid": 0,
    "role": "skulk",
    "command": "string",
    "status": "string",
    "cpuPercent": 0,
    "memoryPercent": 0,
    "rss": {
    "inBytes": 0
    },
    "elapsedSeconds": 0,
    "isChildOfSkulk": false
    }
    ],
    "supervisorRunners": [
    {
    "runnerId": "string",
    "instanceId": "string",
    "nodeId": "string",
    "modelId": "string",
    "deviceRank": 0,
    "worldSize": 0,
    "startLayer": 0,
    "endLayer": 0,
    "nLayers": 0,
    "pid": 0,
    "processAlive": true,
    "exitCode": 0,
    "statusKind": "string",
    "statusSince": "string",
    "secondsInStatus": 0,
    "phase": "created",
    "phaseStartedAt": "string",
    "secondsInPhase": 0,
    "lastProgressAt": "string",
    "activeTaskId": "string",
    "activeCommandId": "string",
    "phaseDetail": "string",
    "lastMlxMemory": {
    "generatedAt": "string",
    "active": {
    "inBytes": 0
    },
    "cache": {
    "inBytes": 0
    },
    "peak": {
    "inBytes": 0
    },
    "wiredLimit": {
    "inBytes": 0
    },
    "source": "string"
    },
    "flightRecorder": [
    {
    "at": "string",
    "phase": "created",
    "event": "string",
    "detail": "string",
    "attrs": {},
    "context": {
    "nodeId": "string",
    "runnerId": "string",
    "pid": 0,
    "instanceId": "string",
    "modelId": "string",
    "rank": 0,
    "worldSize": 0,
    "startLayer": 0,
    "endLayer": 0,
    "nLayers": 0
    },
    "taskId": "string",
    "commandId": "string",
    "mlxMemory": {
    "generatedAt": "string",
    "active": {
    "inBytes": 0
    },
    "cache": {
    "inBytes": 0
    },
    "peak": {
    "inBytes": 0
    },
    "wiredLimit": {
    "inBytes": 0
    },
    "source": "string"
    }
    }
    ],
    "pendingTaskIds": [
    "string"
    ],
    "inProgressTasks": [
    {
    "taskId": "string",
    "taskKind": "string",
    "taskStatus": "string",
    "instanceId": "string",
    "commandId": "string",
    "runnerId": "string",
    "modelId": "string"
    }
    ],
    "completedTaskCount": 0,
    "cancelledTaskIds": [
    "string"
    ],
    "lastTaskSentAt": "string",
    "lastEventReceivedAt": "string",
    "lastEventType": "string",
    "milestones": [
    {
    "at": "string",
    "name": "string",
    "detail": "string"
    }
    ]
    }
    ],
    "placements": [
    {
    "instanceId": "string",
    "modelId": "string",
    "masterNodeId": "string",
    "masterIsPlacementNode": true,
    "localNodeIsPlacementNode": true,
    "placementNodeIds": [
    "string"
    ],
    "runners": [
    {
    "runnerId": "string",
    "nodeId": "string",
    "friendlyName": "string",
    "statusKind": "string",
    "deviceRank": 0,
    "worldSize": 0,
    "startLayer": 0,
    "endLayer": 0,
    "nLayers": 0,
    "isLocal": true,
    "isMaster": true,
    "tasks": [
    {
    "taskId": "string",
    "taskKind": "string",
    "taskStatus": "string",
    "instanceId": "string",
    "commandId": "string",
    "runnerId": "string",
    "modelId": "string"
    }
    ]
    }
    ],
    "warnings": [
    "string"
    ]
    }
    ],
    "dataPlane": {
    "transport": "disabled",
    "reorderBufferEnabled": true,
    "activeStreams": 0,
    "framesReceived": 0,
    "framesDispatched": 0,
    "startedFrames": 0,
    "chunkFrames": 0,
    "completedFrames": 0,
    "failedFrames": 0,
    "cancelledFrames": 0,
    "duplicateFrames": 0,
    "outOfOrderFrames": 0,
    "skippedSequences": 0,
    "lateFrames": 0,
    "missingStartedStreams": 0,
    "missingTerminalStreams": 0,
    "idleTimeouts": 0,
    "transportFailures": 0,
    "firstByteSamples": 0,
    "firstByteSecondsLast": 0,
    "firstByteSecondsAverage": 0,
    "firstByteSecondsMax": 0,
    "streamSpanSamples": 0,
    "streamSpanSecondsLast": 0,
    "streamSpanSecondsAverage": 0,
    "streamSpanSecondsMax": 0,
    "egress": {
    "activeStreamQueues": 0,
    "queueDepth": 0,
    "maxQueueDepth": 0,
    "localShortCircuits": 0,
    "remoteFramesEnqueued": 0,
    "remoteFramesPublished": 0,
    "remoteFramesDropped": 0,
    "remotePublishFailures": 0,
    "remoteBytesPublished": 0,
    "inboundPayloadQueueDepth": 0,
    "inboundPayloadQueueCapacity": 0,
    "inboundTerminalQueueDepth": 0,
    "inboundTerminalQueueCapacity": 0,
    "inboundFramesDropped": 0,
    "idleStreamReclaims": 0,
    "enqueueLatencySamples": 0,
    "enqueueLatencySecondsAverage": 0,
    "enqueueLatencySecondsMax": 0,
    "publishLatencySamples": 0,
    "publishLatencySecondsAverage": 0,
    "publishLatencySecondsMax": 0,
    "owners": {}
    }
    },
    "visionMediaEgress": {
    "activeStreamQueues": 0,
    "queueDepth": 0,
    "maxQueueDepth": 0,
    "localShortCircuits": 0,
    "remoteFramesEnqueued": 0,
    "remoteFramesPublished": 0,
    "remoteFramesDropped": 0,
    "remotePublishFailures": 0,
    "remoteBytesPublished": 0,
    "inboundPayloadQueueDepth": 0,
    "inboundPayloadQueueCapacity": 0,
    "inboundTerminalQueueDepth": 0,
    "inboundTerminalQueueCapacity": 0,
    "inboundFramesDropped": 0,
    "idleStreamReclaims": 0,
    "enqueueLatencySamples": 0,
    "enqueueLatencySecondsAverage": 0,
    "enqueueLatencySecondsMax": 0,
    "publishLatencySamples": 0,
    "publishLatencySecondsAverage": 0,
    "publishLatencySecondsMax": 0,
    "owners": {}
    },
    "visionMediaIngress": {
    "pendingApiCommands": 0,
    "pendingApiBytes": 0,
    "activeApiCommands": 0,
    "activeApiBytes": 0,
    "pendingWorkerAcknowledgements": 0,
    "activeStreams": 0,
    "pendingFrames": 0,
    "retainedBytes": 0,
    "verifiedStreams": 0,
    "pendingFailures": 0,
    "completedStreams": 0,
    "rejectedStreams": 0,
    "expiredStreams": 0
    },
    "provider": {
    "activeStreams": 0,
    "maxActiveStreams": 0,
    "admittedStreams": 0,
    "rejectedStreams": 0,
    "overloadedRejections": 0,
    "inputQueueDepth": 0,
    "maxInputQueueDepth": 0,
    "inputFrames": 0,
    "inputMediaBytes": 0,
    "outputFrames": 0,
    "outputMediaBytes": 0,
    "completedStreams": 0,
    "failedStreams": 0,
    "cancelledStreams": 0,
    "missingTerminalStreams": 0,
    "cancellationRequests": 0,
    "firstOutputSamples": 0,
    "firstOutputSecondsAverage": 0,
    "firstOutputSecondsMax": 0,
    "durationSamples": 0,
    "durationSecondsAverage": 0,
    "durationSecondsMax": 0,
    "activeUnaryCalls": 0,
    "streamSlotsInUse": 0,
    "unaryConcurrencyLimit": 0,
    "streamConcurrencyLimit": 0,
    "capabilities": {}
    },
    "warnings": [
    "string"
    ],
    "tailscale": {
    "running": true,
    "selfIp": "string",
    "hostname": "string",
    "dnsName": "string"
    }
    },
    "error": "string"
    }