Skip to main content

PerformanceEnvelopeSummary

One model+engine+hardware envelope: its per-concurrency curve and knee.

Attributes: hardware_class: Canonical class of the serving node(s), e.g. nvidia-a100-80gb or apple-m4-24gb. model_id: The served model identifier. backend: Resolved engine+backend tag, e.g. vllm-cuda, llama_server-rocm, mlx. quantization: Model quantization label (4bit, Q4_K_M, ...), or empty when unquantized/unknown. buckets: Per-concurrency summaries, ascending by concurrency. knee_concurrency: Concurrency past which aggregate decode throughput stops rising meaningfully, or None when too few concurrency levels have been observed to estimate one. batches: Whether the serving backend decodes concurrent requests together (vLLM). False for the single-stream engines, where the aggregate uses effective concurrency 1. observation_count: Total observations across all buckets.

hardwareClassHardwareclass (string)required
modelIdModelid (string)required
backendBackend (string)required
quantizationQuantization (string)required
buckets object[]required
  • Array [
    • concurrencyConcurrency (integer)required
    requestCountRequestcount (integer)required
    successCountSuccesscount (integer)required
    errorCountErrorcount (integer)required
    ttftSecondsP50 objectrequired
    anyOf
    number
    ttftSecondsP90 objectrequired
    anyOf
    number
    decodeTpsMean objectrequired
    anyOf
    number
    decodeTpsP50 objectrequired
    anyOf
    number
    aggregateDecodeTps objectrequired
    anyOf
    number
  • ]
    • kneeConcurrency objectrequired
    anyOf
    integer
    batchesBatches (boolean)required
    observationCountObservationcount (integer)required
    PerformanceEnvelopeSummary
    {
      "hardwareClass": "string",
      "modelId": "string",
      "backend": "string",
      "quantization": "string",
      "buckets": [
        {
          "concurrency": 0,
          "requestCount": 0,
          "successCount": 0,
          "errorCount": 0,
          "ttftSecondsP50": 0,
          "ttftSecondsP90": 0,
          "decodeTpsMean": 0,
          "decodeTpsP50": 0,
          "aggregateDecodeTps": 0
        }
      ],
      "kneeConcurrency": 0,
      "batches": true,
      "observationCount": 0
    }