ConcurrencyBucketSummary

Aggregated performance at one in-flight-concurrency level.

Attributes: concurrency: Total in-flight requests INCLUDING this one when a request in this bucket was admitted (so a lone request is concurrency 1, and the single-stream engines are always 1). request_count: Observations recorded in this bucket across ALL outcomes (the latency/throughput reservoirs below hold successes only). success_count: Observations that finished cleanly. error_count: Observations that ended in a generation error. ttft_seconds_p50: Median time-to-first-token over the reservoir. ttft_seconds_p90: 90th-percentile time-to-first-token. decode_tps_mean: Mean steady-state decode tokens/second. decode_tps_p50: Median steady-state decode tokens/second. aggregate_decode_tps: The instance's total useful decode throughput at this concurrency, the quantity whose knee an admission controller targets. For a batching backend it is concurrency * decode_tps_mean; for a serial (single-stream) backend, where concurrent client requests queue rather than decode together, the effective concurrency is 1 (= decode_tps_mean) so the knee is not biased upward.

concurrencyConcurrency (integer)required

requestCountRequestcount (integer)required

successCountSuccesscount (integer)required

errorCountErrorcount (integer)required

ttftSecondsP50 objectrequired

anyOf

• number
• null

number

ttftSecondsP90 objectrequired

anyOf

• number
• null

number

decodeTpsMean objectrequired

anyOf

• number
• null

number

decodeTpsP50 objectrequired

anyOf

• number
• null

number

aggregateDecodeTps objectrequired

anyOf

• number
• null

number

ConcurrencyBucketSummary

{
  "concurrency": 0,
  "requestCount": 0,
  "successCount": 0,
  "errorCount": 0,
  "ttftSecondsP50": 0,
  "ttftSecondsP90": 0,
  "decodeTpsMean": 0,
  "decodeTpsP50": 0,
  "aggregateDecodeTps": 0
}