Performance Tuning

SIE provides several tuning parameters that affect throughput, latency, and resource usage. This guide covers the main configuration options.

Batching Parameters

Batching groups requests to maximize GPU utilization. The tuning surface depends on deployment mode:

Mode	Runtime that forms batches	Primary knobs
Standalone sie-server	Python sie-server	SIE_MAX_BATCH_WAIT_MS, SIE_MAX_BATCH_REQUESTS, per-model max_batch_tokens
Kubernetes queue mode	SIE server sidecar inside the worker pod	workers.common.workerSidecar.batcher.*, pipelineDepth, adaptive.*

max_batch_cost

Maximum total cost per batch. For text, cost equals token count. Default: 16384 tokens.

Batch cost is an internal default in BatchConfig and is configured per-model, not via environment variable.

max_batch_wait_ms

Maximum time to wait for more requests before processing a batch. Default: 10ms.

# Environment variable
export SIE_MAX_BATCH_WAIT_MS=20

Lower values reduce latency for sparse traffic. Higher values improve batching efficiency under load.

max_batch_requests

Maximum number of requests per batch. Default: 64.

# Environment variable
export SIE_MAX_BATCH_REQUESTS=128

This is a secondary limit. Cost-based batching typically triggers first for text workloads.

Queue-mode Sidecar Defaults

In Kubernetes, the SIE server sidecar pulls from JetStream and sends fully formed RunBatch IPC calls to the sie-server adapter. The production defaults are intentionally conservative:

Helm value	Default	Purpose
workers.common.workerSidecar.pipelineDepth	2	One Python batch active and one queued behind it
workers.common.workerSidecar.batcher.coalesceMs	5	Server-sidecar batch coalesce window
workers.common.workerSidecar.batcher.maxBatchRequests	12	Hard item cap per SIE server sidecar batch
workers.common.workerSidecar.adaptive.minQuantumMs	2	Pull-loop coalesce floor
workers.common.workerSidecar.adaptive.maxQuantumMs	15	Pull-loop coalesce ceiling
workers.common.workerSidecar.adaptive.targetP50Ms	50	Pull-loop latency target

Treat these as a group. Raising maxBatchRequests without checking pipelineDepth and adaptive wait can improve throughput while hurting p50 and p95 latency.

Tuning Strategy

For low-latency Docker workloads, reduce SIE_MAX_BATCH_WAIT_MS to 5ms or less. For high-throughput Docker workloads, increase SIE_MAX_BATCH_WAIT_MS and SIE_MAX_BATCH_REQUESTS.

For Kubernetes queue-mode workloads, start with the Helm SIE server sidecar defaults. Increase batcher.maxBatchRequests only after sie_worker_scheduler_batch_items, sie_worker_backend_process_seconds, and sie_gateway_request_latency_seconds show that the GPU is underfed and latency has room.

Memory Thresholds

SIE uses reactive LRU eviction to manage GPU memory. No static VRAM budget is required.

Pressure Threshold

When memory usage exceeds this percentage, the least-recently-used model is evicted. Default: 85%.

# Environment variable
export SIE_MEMORY_PRESSURE_THRESHOLD_PERCENT=85

Lower values keep more headroom for inference spikes. Higher values allow more models to stay loaded.

How Eviction Works

The memory manager checks pressure at two points:

1. Before loading: If above threshold, evict LRU model first
2. After each batch: Background check for gradual memory growth

Models are tracked by last-use time. The oldest model is evicted first.

# From memory.py - LRU tracking
def touch(self, model_name: str) -> None:
    if model_name in self._models:
        self._models[model_name].touch()
        self._models.move_to_end(model_name)

Device-Specific Behavior

Memory tracking adapts to your hardware:

Device	Memory Source
CUDA	NVML device memory query
MPS	PyTorch allocated memory
CPU	System RAM via psutil

Attention Backend

The attention implementation affects inference speed significantly.

Available Backends

Backend	Requirements	Speedup
flash_attention_2	Ampere+ GPU, flash-attn package	2-4x
sdpa	PyTorch 2.0+	1.5-2x
eager	Any	Baseline

Configuration

# Auto-select best available (default)
export SIE_ATTENTION_BACKEND=auto

# Force specific backend
export SIE_ATTENTION_BACKEND=flash_attention_2
export SIE_ATTENTION_BACKEND=sdpa

Auto mode selects Flash Attention 2 if available, then SDPA, then eager.

Flash Attention Requirements

Flash Attention 2 requires:

• CUDA compute capability 8.0+ (Ampere: A100, RTX 30xx, RTX 40xx)
• The flash-attn package installed
• FP16 or BF16 compute precision (not FP32)

If requirements are not met, the server uses SDPA automatically.

Compute Precision

Control the precision used for model inference:

# Options: float16, bfloat16, float32
export SIE_DEFAULT_COMPUTE_PRECISION=float16

Precision	Memory	Speed	Compatibility
float16	Low	Fast	All CUDA GPUs
bfloat16	Low	Fast	Ampere+, MPS, CPU
float32	High	Slow	All devices

BF16 offers better numerical stability than FP16 for some models. FP32 is mainly for debugging.

Preprocessing Workers

Tokenization and image processing run in a CPU thread pool.

# Environment variable
export SIE_PREPROCESSOR_WORKERS=8

Default: 4. Increase for high request rates. Decrease on memory-constrained systems.

The thread pool is shared across all models. Both tokenization and image preprocessing use the same pool.

Environment Variables

The most commonly used tuning parameters can be set via environment variables with the SIE_ prefix:

Variable	Default	Description
SIE_MAX_BATCH_REQUESTS	64	Max requests per batch
SIE_MAX_BATCH_WAIT_MS	10	Max wait time (ms)
SIE_MAX_CONCURRENT_REQUESTS	512	Request queue size
SIE_RUST_PIPELINE_DEPTH	2	Queue-mode SIE server sidecar IPC pipeline depth
SIE_BATCHER_COALESCE_MS	5	Queue-mode SIE server sidecar batch coalesce window
SIE_BATCHER_MAX_BATCH_REQUESTS	12	Queue-mode SIE server sidecar max items per batch
SIE_ADAPTIVE_MIN_QUANTUM_MS	2	Queue-mode pull-loop wait floor
SIE_ADAPTIVE_MAX_QUANTUM_MS	15	Queue-mode pull-loop wait ceiling
SIE_ADAPTIVE_TARGET_P50_MS	50	Queue-mode pull-loop latency target
SIE_MEMORY_PRESSURE_THRESHOLD_PERCENT	85	Eviction trigger (%)
SIE_PREPROCESSOR_WORKERS	4	CPU thread pool size
SIE_ATTENTION_BACKEND	auto	Attention implementation
SIE_DEFAULT_COMPUTE_PRECISION	float16	Model precision

Benchmarking Changes

Use the eval runner to measure the impact of tuning changes:

# Performance benchmark
mise run eval BAAI/bge-m3 -t mteb/NFCorpus --type perf -s sie

# Compare before/after
mise run eval BAAI/bge-m3 -t mteb/NFCorpus --type perf -s sie,targets

The perf eval reports throughput (items/sec), latency percentiles, and GPU utilization.

See the Evals documentation for the full benchmarking workflow.

What’s Next

• Request Lifecycle - how batching and memory work together
• Evals - benchmark your configuration changes