Skip to main content

AssessedReplica

One independent model replica in the conservative concurrency plan.

shardingSharding (string)required
shards object[]required
  • Array [
    • nodeIdNodeid (string)required
    backend objectrequired
    anyOf
    string
    estimatedMemoryBytes objectrequired

    Shared estimated footprint; null for runtime-selected RPC partitions.

    anyOf
    integer
    usableMemoryBytesUsablememorybytes (integer)required

    Memory available after current commitments and working-set caps.

    memoryPoolMemorypool (string)required

    Possible values: [ram, vram]

  • ]
    • concurrentRequestsConcurrentrequests (integer)required
    contextPoolTokensContextpooltokens (integer)required

    Aggregate reserved context for requests assigned to this replica.

    existingInstanceId object
    anyOf
    string
    AssessedReplica
    {
      "sharding": "string",
      "shards": [
        {
          "nodeId": "string",
          "backend": "string",
          "estimatedMemoryBytes": 0,
          "usableMemoryBytes": 0,
          "memoryPool": "ram"
        }
      ],
      "concurrentRequests": 0,
      "contextPoolTokens": 0,
      "existingInstanceId": "string"
    }