SIE vs vLLM: Which Should You Deploy for Multi-Model Workloads?

Short version: vLLM serves one model per server, and it is very good at that. SIE (Superlinked Inference Engine) serves every model an agent calls from one cluster: the embedding model, the reranker, the guard models, the extractor and the LLM, behind one API. It runs SGLang inside for the models that suit it, so choosing SIE does not mean giving up that class of runtime. Run vLLM directly when one flagship model carries your traffic and you want its engine flags in your own hands. Run SIE when the agent calls several models and you would otherwise run a vLLM server for each.

One vLLM server serves one model

vLLM’s own docs put it directly: each vLLM instance supports one model runner. To serve five models you run five processes and put a router in front of them. Each process reserves its own slice of GPU memory through --gpu-memory-utilization, which defaults to 0.92 (engine arguments). Two servers on one GPU means splitting that by hand, 0.5 each, and an idle model keeps its half.

vLLM does have a way to park a model: sleep mode offloads weights to CPU memory or drops them. It is opt-in (--enable-sleep-mode), the /sleep and /wake_up endpoints need VLLM_SERVER_DEV_MODE=1, and something outside vLLM has to decide when to call them. That something is the scheduler you would be writing.

None of this is a flaw. vLLM is an engine for running one model fast, and it does that job well.

What SIE does instead

SIE is a cluster for running many models. You assign each model to a pool; each pool runs on its own workers and GPUs. Inside a pool, a worker loads a model the first time a request needs it, keeps the busy ones resident and evicts the least recently used model when GPU memory passes 95%. Models you pin are never evicted. Every model reserves memory sized to its weights and KV budget, so two guard models such as Qwen3Guard 4B and Granite Guardian 2B share one GPU instead of claiming one each.

SIE also keeps LLMs and encoders apart. A configuration that puts a generating model in the same pool as embedding, scoring or extraction models is refused at load time. Your LLM’s GPUs, and its KV cache, belong to the models you chose to put next to it.

Under the hood, SIE runs SGLang for the LLMs and some embedding models, and its own adapters for encoders, rerankers, GLiNER extractors, OCR and speech models. Each model’s profile names the adapter it runs on, so the backend is a reviewed choice per model rather than a runtime guess. Large models span GPUs with tensor parallelism: Gemma 4 31B runs on one H100, Qwen3.5 122B-A10B on two 80 GB GPUs, and GLM-5.3 Flash on eight.

Not every model an agent calls is an LLM

vLLM’s pooling API covers more than chat: embeddings, classification, cross-encoder scoring and token classification all run on it (pooling models). What it does not run is the long tail an agent picks up over time. GLiNER-style extractors have no upstream vLLM support, and each new model family still means another vLLM process with its own memory share and its own endpoint shape.

SIE puts all of them behind five operations: encode, score, extract, generate and decide, with OpenAI-compatible routes for chat and embeddings. Adding a model from the catalog is a configuration change, not a new service.

LoRA adapters or separate models

LoRA fine-tunes one base model for several similar jobs, such as support answers tuned for each product line. If your jobs look like that, adapters are the cheaper option and you should use them.

vLLM serves adapters and batches requests for different ones together, up to --max-loras, which defaults to 1 (LoRA docs). Loading a new adapter while the server runs goes through /v1/load_lora_adapter, behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, and the docs advise against enabling it in production outside an isolated, trusted environment.

For SGLang generation in SIE, declare adapters in the model’s profile (lora_paths) and select one per request (lora_adapter). By default, all declared adapters load at startup and stay pinned.

Enable on-demand loading with preload_loras: [] and the capacity settings in the runtime LoRA guide. SIE then loads adapters on first use and evicts the least recently used unpinned adapter when its slots fill. A nonempty preload_loras list preloads and pins selected adapters while leaving the rest on demand. Generation requests for different adapters can share a batch. SGLang embedding profiles use the same loading choices and group requests by the selected adapter.

What LoRA does not do is turn your LLM into an embedding model, a reranker or an extractor. Those are different kinds of model, usually under 1B parameters, built for exactly those jobs.

LLM requests for different adapters shared batches

We ran runtime LoRA on one L4 with SGLang 0.5.10.post1, first for generation with Qwen3-0.6B, then for embeddings with Qwen3-Embedding-0.6B. Each model declared five untrained rank-4 test adapters, one of them pinned, with room for three ready adapters. Generation ran 22 forward passes that mixed adapters A and B in one batch, each adapter in its own GPU slot.

The first request for a cold adapter got a 503 while SIE loaded it, and a later request succeeded. An adapter the model had not declared got a 400. As requests cycled through the adapters, SIE evicted and reloaded the unpinned ones on demand, kept the pinned one ready and never restarted the engine.

Switching back to the base embedding model returned its warmed-up vectors exactly: the largest difference was 0. Adapters A and B produced different vectors, with a largest difference of 0.1061. Generation returned to the same warmed-up text and observed token probabilities.

The measurement record has the model versions and observations.

Where vLLM wins

• One flagship model carries nearly all the traffic and has the GPUs to itself. There is little to share, and SIE’s model management adds a layer you do not need.
• You want every engine flag yourself: speculative decoding settings, custom kernels, a build from main for a model released yesterday.
• Your platform is already vLLM end to end, with the production stack router, and you have no models outside its coverage.

At a glance

What it looks like in code

One client, one endpoint, every model named per request:

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

vectors = client.encode("BAAI/bge-m3", Item(text=user_query))
ranked = client.score(
    "BAAI/bge-reranker-v2-m3",
    Item(text=user_query),
    [Item(text=c) for c in candidates],
)
fields = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=document),
    labels=["party", "date", "amount"],
)
answer = client.generate(
    "Qwen/Qwen3.8-27B-FP8",
    build_prompt(ranked, fields),
    max_new_tokens=512,
    lora_adapter="support-billing",  # declared in the model's lora_paths
)

Where to deploy

The same engine and model IDs run in three places, so the choice is operational (deployment docs). On SIE Cloud, Superlinked runs the cluster and you get an API key. In your own cloud, Terraform modules for AWS, GCP, Azure and Alibaba Cloud create the cluster, Helm installs into any Kubernetes cluster you already run, and each pool can scale down to zero when idle. Locally, one container on an NVIDIA GPU serves the same SDK calls.

FAQ

Do we still need vLLM if we run SIE? Not for the models SIE serves. SIE runs SGLang inside for LLMs, so you keep that runtime; what goes away is the fleet of single-model servers and the memory budgets between them.

Will the small models take memory from our LLM? Only if you put them in its pool, and SIE refuses to put embedding, scoring or extraction models there at all. Pin the LLM and nothing evicts it.

Is SIE faster than vLLM on one model? We have not published a matched comparison of one model on both engines, and you should not pick on that basis. Pick on how many models you serve and who manages their memory.

Can I A/B test two encoders without a second deployment? Yes. Name the model per request against one endpoint; the second one loads on its first request.

Does SIE compete with SGLang? It uses it. SGLang is the runtime under SIE’s LLMs; SIE adds the pools, model lifecycle and task API around it.

Star the repo on GitHub · Get started with SIE Cloud · Sharing GPUs between models · Browse the model catalog