vLLM vs SGLang vs SIE for Production Search Pipelines

Short version: a production search request passes through four hops, and each hop calls a different model. vLLM and SGLang are close rivals as engines; each runs one model per server, so a four-hop pipeline built on them means four or more servers. SIE (Superlinked Inference Engine) serves all four hops from one cluster, the retrieval encoders, the reranker, the extractor and the LLM, and runs SGLang inside it for the generation step. The comparison only looks like a three-way choice when the pipeline is drawn as one box.

So rather than rank them, trace a request.

A production search request has four hops

Ask a real question of a real corpus and here is what has to happen.

Retrieve. The query gets encoded and matched against an index. Past a prototype this is rarely one encoder: dense embeddings catch meaning, sparse embeddings catch exact terms the dense vector blurs, and multi-vector late interaction keeps token-level detail a single vector compresses out. Each is a separate model.

Rerank. The shortlist goes to a cross-encoder, which scores query and document together. In most pipelines this is the biggest precision gain, and it is another model class.

Extract. Before the answer is written, the retrieved documents often need structure pulled out: entities, fields, a schema-valid object, the text of a scanned page. More models.

Generate. An LLM writes the answer from what came back.

Four hops, somewhere between four and ten distinct models before you add anything clever.

Hop by hop, which model and which operation

Both engines cover more than generation. vLLM’s pooling models run embeddings, classification and cross-encoder scoring; SGLang serves embedding and reranker models too, though a single SGLang server does either embedding or generation, not both. The pattern in the right-hand column is the cost: one process per model, each with its own memory share, endpoint shape and autoscaler.

vLLM and SGLang compete with each other

They are rival engines for running a model fast, especially an LLM. Both do continuous batching and paged KV memory, and the choice between them comes down to model family support, long-context behavior, structured output handling and how each performs on the weights you plan to serve. Benchmark both on your model and your hardware; this article will not invent numbers for that decision.

SIE is not a third engine in that race. It runs SGLang inside for its LLMs and some embedding models, and its own adapters for the encoders, rerankers and extractors. Each model’s profile names the adapter it runs on. So “vLLM vs SGLang vs SIE” is partly a category error: SGLang can be inside SIE, doing the generation hop.

The four hops are one workload

What decides the architecture is that the four hops share a traffic shape. Retrieval runs on every query; reranking on every query that returns enough candidates; extraction only for some documents; generation once per answer. Deployed one model per server, each of those servers is sized for its own peak and idles the rest of the time.

SIE serves them as one cluster. Models live in pools, each on its own workers and GPUs. A worker loads a model on its first request, keeps busy models resident and evicts the least recently used one when GPU memory passes 95%; pinned models never leave. Each model reserves memory sized to its weights and KV budget, so small models share a GPU. The LLM goes in a pool of its own, because SIE never mixes generation models with encoders in one pool.

Batching works across the pool rather than per server. A stateless gateway puts requests on a shared queue, and a sidecar in each worker pulls from it and forms the batch before the GPU runs, so a worker batches everything waiting for its model rather than only what a router sent it.

Serving dense, sparse, multi-vector and rerank models from one deployment is the case we wrote up separately.

Per-hop model choice is the real lever

Once the hops are separate you can pick the model per hop by quality, latency and cost, and change it with a config line. The task pages carry the measured comparisons, with the dataset, hardware and rival named for each: search, hybrid search, rerank, named entities and chat.

A reference pipeline

All four hops on one client (vector DB and prompt assembly left as placeholders):

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
query = "late delivery penalty"

# Hop 1: retrieve. Dense and sparse from one endpoint, named per request.
dense = client.encode("BAAI/bge-m3", Item(text=query))
sparse = client.encode("prithivida/Splade_PP_en_v2", Item(text=query), output_types=["sparse"])
candidates = vector_db.hybrid_search(dense, sparse, limit=100)
candidate_text = {c["id"]: c["text"] for c in candidates}

# Hop 2: rerank the shortlist with a cross-encoder.
ranked = client.score(
    "BAAI/bge-reranker-v2-m3",
    Item(text=query),
    [Item(id=c["id"], text=c["text"]) for c in candidates],
)
top_ids = [entry["item_id"] for entry in ranked["scores"][:10]]

# Hop 3: extract structure from what came back.
fields = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=candidate_text[top_ids[0]]),
    labels=["organisation", "date", "amount"],
)

# Hop 4: generate the answer on the same cluster.
top_passages = [candidate_text[i] for i in top_ids]
answer = client.generate(
    "Qwen/Qwen3.8-27B-FP8",
    build_prompt(top_passages, fields),
    max_new_tokens=512,
)

One client, one endpoint, models named per request.

Where pipelines actually break

The failure is rarely the wrong generation engine. It is that retrieval, reranking, extraction and generation were built as separate deployments, each pinned to its own GPU, each scaled on its own, each with its own client and monitoring. The cost shows up as idle GPUs, and the velocity cost shows up every time someone wants to A/B test a second reranker and has to provision hardware for it.

When to run vLLM or SGLang directly

• One flagship LLM carries nearly all your traffic and has its GPUs to itself.
• You need an engine flag or a fresh model build that SIE does not expose yet.
• The LLM is so large that it needs prefill and decode split across nodes; put it on llm-d or Dynamo and serve the other hops on SIE.

FAQ

Should I use vLLM or SGLang for my generation hop? If you run the LLM yourself outside SIE, benchmark both on your model and hardware; they are close peers. On SIE, the generation hop runs on SGLang.

Does SIE replace vLLM or SGLang in a search pipeline? It replaces the per-model servers. SIE serves retrieval, reranking, extraction and generation from one cluster, with SGLang underneath the LLM.

Why not run my encoders on vLLM to keep one system? You can run several, one server per model, each with its own memory share. You still need a separate service for GLiNER-style extraction, and an idle encoder keeps its memory.

Do I need all three retrieval modes? No. Most pipelines start with dense only. Serving them from one deployment means adding sparse or multi-vector later is a config change rather than a new deployment.

Can I keep my vector database? Yes. SIE produces the vectors and your database indexes them, with documented integrations for Chroma, Qdrant, Weaviate and LanceDB.

Star the repo on GitHub · Get started with SIE Cloud · Read the encode docs · Browse the model catalog