Model Adapters

Adapters connect model families to the inference engine. An adapter is the runtime implementation of a model family’s forward pass: it loads the weights onto a device, prepares and batches inputs, runs the model on a specific compute backend (Flash Attention, SGLang, or PyTorch SDPA), and shapes the raw outputs into the engine’s output types (dense, sparse, multivector, score, or JSON). Every adapter implements the same lifecycle protocol, which lets SIE serve 80+ models with consistent behavior.

Deployments can configure and extend adapters, so the set below varies per installation. For the current set, browse packages/sie_server/src/sie_server/adapters/ in the repo.

What Are Adapters

An adapter wraps a specific model architecture or library. It handles:

• Loading model weights onto a device (CPU, CUDA, MPS)
• Inference via encode(), score(), or extract() methods
• Unloading with proper memory cleanup

One adapter can serve many models. For example, SentenceTransformerDenseAdapter works with all-MiniLM, E5, BGE, and hundreds of other compatible models.

Adapter Protocol

Every adapter exposes the same core lifecycle:

• Capabilities to declare input/output support
• Dimensions for output shapes
• Load/Unload for device placement and cleanup
• Encode/Score/Extract for inference

Capabilities

Each adapter declares its capabilities:

Field	Type	Description
inputs	list[str]	Supported input modalities: “text”, “image”, “audio”, “video”, “document”
outputs	list[str]	Output types: “dense”, “sparse”, “multivector”, “score”, “json”, “tokens”

Capabilities are static metadata in the model config and adapter implementation. Task support (encode, score, extract) is declared in the model YAML tasks block, not through capability flags.

Dimensions

Adapters report output dimensions for validation and client usage:

Field	Description
dense	Dense vector dimensionality (e.g., 1024)
sparse	Vocabulary size for sparse vectors
multivector	Per-token embedding dimension

Compute Engines

Adapters use different compute backends depending on model architecture:

Flash Attention 2

Flash Attention with variable-length sequences eliminates padding waste. Uses flash_attn_varlen_func to pack sequences and process without padding tokens.

Benefits:

• Higher throughput (no wasted compute on padding)
• Lower memory usage (no padded tensors)

SGLang

SGLang provides memory-efficient inference for large LLM embedding models (4B+). Pre-allocates KV cache to prevent OOM under concurrent load.

Benefits:

• Stable memory usage with concurrent requests
• Handles 4B-8B parameter models reliably
• LoRA adapter support via HTTP API

PyTorch with SDPA

Standard PyTorch with Scaled Dot-Product Attention. Uses native transformers libraries like sentence-transformers.

Benefits:

• Broadest compatibility
• Works on CPU, CUDA, and MPS
• Simple debugging

Adapter Catalog

Two sources track the full set of adapters and the models each one serves, and both stay current as models change:

• Adapters and their implementations: packages/sie_server/src/sie_server/adapters/ in the repo, grouped by family (dense, sparse, ColBERT, reranker, vision, extraction).
• Supported models, filterable by task and modality: Model Catalog.

Memory Management

Adapters must fully release GPU memory in unload() so LRU eviction is safe.

    if self._model is not None:
        del self._model
        self._model = None

    self._device = None

    # Release GPU memory
    import gc
    gc.collect()
    if device and device.startswith("cuda"):
        torch.cuda.empty_cache()
    elif device == "mps":
        torch.mps.empty_cache()

The registry tracks memory usage via memory_footprint() for LRU eviction.

LoRA Support

Some adapters support dynamic LoRA adapter loading:

def supports_lora(self) -> bool:
    """Return True if this adapter supports LoRA."""
    ...

def load_lora(self, lora_path: str, revision: str | None = None) -> int:
    """Load a LoRA adapter, return memory usage."""
    ...

def set_active_lora(self, lora_name: str | None) -> None:
    """Switch active LoRA before inference."""
    ...

SGLang adapters use the HTTP API for LoRA switching. PEFT-based adapters use the PEFTLoRAMixin for in-process loading.

Writing Custom Adapters

For adding support for new model architectures, see Adding Models.

The typical workflow:

1. Identify the model architecture (BERT, Qwen2, custom)
2. Choose a compute backend (SDPA, Flash, SGLang)
3. Implement the adapter protocol
4. Create a model config in packages/sie_server/models/

What’s Next

• Adding Models - configure new models
• Model Catalog - all supported models