InferenceWorkload

Requested text workload; context includes input and generated tokens.

contextTokensContexttokens (integer)

Input plus output context tokens per simultaneous request.

Possible values: >= 256 and <= 1048576

Default value: 8192

concurrentRequestsConcurrentrequests (integer)

Requested simultaneous serving width, without a performance SLO.

Possible values: >= 1 and <= 16

Default value: 1

InferenceWorkload

{
  "contextTokens": 8192,
  "concurrentRequests": 1
}