llm-d vs Dynamo vs SIE: Multi-Model Serving Compared

Short version: all three sit above the inference runtime, so they are fair to compare. llm-d and NVIDIA Dynamo make one large LLM fast across many GPUs: they split prefill and decode onto separate workers and route requests by what is already in each worker’s KV cache. SIE (Superlinked Inference Engine) serves every model an agent calls from one cluster, the embedding model, reranker, guard models, extractor and LLM, and loads each into a pool’s GPUs as its traffic arrives. Neither llm-d nor Dynamo swaps models in and out of a GPU, and neither serves GLiNER-style extraction. SIE does not split prefill and decode. Pick by the shape of your traffic.

All three are cluster layers, not runtimes

vLLM, SGLang and TensorRT-LLM are runtimes: they execute a model on hardware. llm-d, Dynamo and SIE are the layer above, the one that decides which worker gets which request, how work is batched, when capacity scales and what happens when a model is not loaded.

Dynamo says so in its own README: it sits above engines like vLLM, SGLang and TensorRT-LLM and turns a cluster of GPUs into a coordinated system. llm-d is built on vLLM and plugs into Kubernetes through the Gateway API Inference Extension; KServe’s LLMInferenceService now builds on it too. SIE runs SGLang for its LLMs and its own adapters for encoders, rerankers, extractors, OCR and speech, with each model’s profile naming the adapter it uses.

llm-d and Dynamo optimize one large model

Both projects start from the same observation. A generative request has two phases with different hardware appetites: prefill is compute-bound, decode is memory-bandwidth-bound, and running both on one worker leaves each starving the other. Split them onto separate GPU pools and each pool saturates the resource it needs.

Everything else follows from that split. Both move KV cache between prefill and decode workers with NVIDIA’s NIXL library. Both route with cache awareness: llm-d’s inference scheduler scores decode pods by prefix match length, and Dynamo’s KV-aware router weighs queue depth against cache overlap. Dynamo adds a Planner that resizes prefill and decode pools against a latency target and a KV Block Manager that tiers cache to CPU memory and storage. llm-d adds SLO-aware scheduling and KEDA-based scaling down to zero.

The published gains are real, and they are gains on large-model traffic. From partner engineering blogs: up to 70% higher tokens per second from disaggregation against standard vLLM on GPT-OSS on B200 hardware, 10 to 30% higher throughput for GPT-OSS-120B and Llama 3.3 70B on MI300X, and around 40% lower time to first token with predicted-latency scheduling.

Both projects are candid about where that stops paying. llm-d’s disaggregation guide recommends the pattern for large models, long inputs and sparse mixture-of-experts architectures. An encoder call has no decode phase at all.

Multiple models are possible, but static. An llm-d InferencePool shares one base model and one model server. Dynamo’s many-model serving guide packs two Qwen3-4B models onto each A100-40GB with HAMi or KAI GPU fractions: a fixed placement, decided at deploy time, with no model coming or going as traffic shifts.

SIE serves the whole set of models

An agent’s traffic is uneven across its models. The embedding model runs on every query, the guard model twice per conversation, the OCR model when a scanned document arrives. SIE is built around that unevenness.

Models live in pools, and each pool runs on its own workers and GPUs. A worker loads a model the first time a request needs it, keeps the busy ones resident and evicts the least recently used one when GPU memory passes 95%; pinned models never leave. Each model reserves memory sized to its weights and KV budget, so small models share a GPU. LLMs never share a pool with embedding, scoring or extraction models, because SIE refuses that configuration.

Requests go through a stateless gateway onto a shared queue on NATS JetStream. A sidecar inside each worker pulls from the pool’s queue and forms the batch before the GPU runs, so a worker batches across everything waiting for its model rather than only what a router happened to send it. Each pool scales with its queue through KEDA, down to zero when idle.

Large models fit too. SIE runs tensor parallelism across up to eight GPUs on its SGLang adapters: Gemma 4 31B on one H100, Qwen3.5 122B-A10B on two 80 GB GPUs, GLM-5.3 Flash on eight.

At a glance

The deciding question

Count two things: how many distinct models one user-facing request touches, and how large the largest of them is.

One model, very large, at high concurrency across nodes: that is llm-d and Dynamo territory. Disaggregation and KV-aware routing are why you go there, and SIE does not do either.

Several models with uneven traffic, the largest of which fits on up to eight GPUs per replica: that is SIE. A contract review agent with nine models, from a guard model to an OCR reader to an LLM, is one application; an organization runs many of them, and its cluster ends up serving dozens of models at once. Pinning one model per GPU is then the whole bill.

A mixed fleet

Some stacks need both. If one frontier-scale LLM carries most of your tokens across many nodes, run it on Dynamo or llm-d and run every other model on SIE: the embedding, guard, extraction and smaller generation models. The two talk over HTTP and neither needs to know how the other schedules.

If your largest model fits SIE’s pools, one cluster serves everything and you skip the second control plane.

Try it on your own traffic

SIE runs locally with one command, so you can profile your real model mix before committing to a cluster:

pip install "sie-server[local]" && sie-server serve

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
client.encode("BAAI/bge-m3", Item(text="profile this against your own model mix"))

FAQ

Is SIE a replacement for llm-d or Dynamo? For one large LLM spread across many nodes, no: SIE does not split prefill and decode. For an agent’s full set of models, including LLMs that fit on up to eight GPUs, SIE serves all of them from one cluster.

Can I run SIE alongside Dynamo or llm-d? Yes. They are separate control planes over separate GPU pools, reached over HTTP.

Do llm-d or Dynamo serve embeddings and reranking? Dynamo added an embedding worker pool in v1.5; llm-d is built for generation. Neither serves GLiNER-style extraction, and neither loads and evicts models with traffic.

Which is the safer choice if our workload is still changing? Agent workloads tend to add models rather than grow one, because each new capability is usually a new specialist. A cluster that loads models on demand absorbs that without a new deployment per model.

Are these numbers comparable head to head? No. The llm-d figures come from those projects and their partners, on large-model benchmarks. We have not published a matched SIE run against them, and this article does not present one.

Star the repo on GitHub · Get started with SIE Cloud · Sharing GPUs between models · Browse the model catalog