Performance Evaluation

SIE gives you two vantage points on performance: per-request timing that arrives with every encode response, and cluster-wide dashboards fed by OpenTelemetry. Between them you can tell whether time went to queueing, tokenization, or the GPU, and whether the cluster keeps up as load grows.

Request Timing Metadata

Encode responses carry a server-side timing breakdown when the server tracked one, and the SDK attaches it to each result:

Python

from sie_sdk import SIEClient

client = SIEClient("http://your-sie-endpoint:8080")
result = client.encode("BAAI/bge-m3", {"text": "Hello world"})
print(result["timing"])

TypeScript

import { SIEClient } from "@superlinked/sie-sdk";

const client = new SIEClient("http://your-sie-endpoint:8080");
const result = await client.encode("BAAI/bge-m3", { text: "Hello world" });
console.log(result.timing);

Field	Meaning
total_ms	Total end-to-end time
queue_ms	Time waiting in queue
tokenization_ms	Time tokenizing input
inference_ms	GPU forward pass time
postprocessing_ms	Postprocessor transforms time

The table shows the wire field names. The Python SDK’s typed TimingInfo covers total_ms, queue_ms, tokenization_ms, and inference_ms; postprocessing_ms is omitted from the wire when it is 0 and has no typed field in the SDK. The TypeScript SDK exposes a four-field camelCase subset: totalMs, queueMs, tokenizationMs, inferenceMs.

Score responses return scores and token usage without a timing block, so time reranking calls client-side.

Server Time vs Wall Time

Client-side wall time adds network transfer and any SDK retries on top of total_ms. The SDK retries transparently while capacity provisions or a model cold-loads, so a single slow call is often a scale-up event rather than slow inference. In the Python SDK, check client.last_retry_count after a call to see how many retries it performed, and keep cold-start runs out of your steady-state numbers.

Isolating a Benchmark With Pools

Latency measured while production traffic shares the GPUs is a measurement of contention, not of the model. Resource pools reserve exclusive capacity for the duration of a benchmark:

Python

client.create_pool("eval", {"l4": 2})
result = client.encode("BAAI/bge-m3", {"text": "Hello"}, gpu="eval/l4")
client.delete_pool("eval")

TypeScript

await client.createPool("eval", { l4: 2 });
const result = await client.encode("BAAI/bge-m3", { text: "Hello" }, { gpu: "eval/l4" });
await client.deletePool("eval");

Load Patterns

The public repo does not ship a load-testing harness. Drive load with an asyncio script against the SDK, or any HTTP load generator you already use. Two patterns cover most questions:

• Constant rate. Hold a fixed request rate for several minutes and record latency percentiles. This is your steady-state number for capacity planning.
• Ramp. Increase the rate stepwise until p99 degrades. The point where queue_ms starts climbing while inference_ms stays flat is saturation; the fix at that point is more replicas, not a faster model.

Run each measurement long enough to outlast autoscaling, and separate results taken during scale-up from results at stable worker counts.

Cluster Dashboards

The Helm chart ships seven Grafana dashboards in deploy/helm/sie-cluster/files/dashboards/, provisioned automatically when the bundled monitoring stack is enabled:

• SIE Cluster Overview
• SIE Model Performance
• SIE Perf Tuning (L4)
• SIE Queue Routing
• SIE Worker Health
• SIE Tracing
• SIE Generation

For benchmark runs, SIE Model Performance is the one to watch: per-model QPS, p95 latency, average batch size, and a latency breakdown by operation. Setup and example PromQL queries are in Monitoring.

What’s Next

• Quality Evaluation - measure retrieval accuracy with NDCG@10
• Monitoring & Observability - the full metrics and dashboard stack
• SDK Reference - client options for timeouts and GPU routing