Self-hosted inference for enterprise AI

Not as separate deployments. SIE runs SGLang inside the cluster for the models that suit it, so you keep its speed. What goes away is the fleet around it: a server per model, each with its own API, memory share and autoscaler.

Only if you put them in its pool, and SIE refuses to put embedding, scoring or extraction models there at all. Pin the LLM and SIE never evicts it. Run several vLLM servers on one GPU instead and you split its memory by hand, while an idle model keeps its share.

SIE tries to recover before it fails: it frees cached memory, unloads the least recently used model and splits the batch, then returns a retryable error if memory is still short. Each SGLang model runs in its own process; if one crashes, SIE unloads it and reloads it when the next request arrives.

Those decide how much of a GPU each pod gets; none of them knows which model a request needs. SIE works at the model level: it loads a model when its requests arrive and unloads the least recently used one when memory runs short. Use MIG when tenants need a hardware boundary between them.

When one large LLM carries most of your traffic across many GPUs: they split prefill and decode onto separate GPUs and route requests by KV cache. Neither swaps models in and out of a GPU or serves GLiNER-style extraction models, and SIE does both.

Yes, when several jobs can share one base model, like support answers tuned for each product line. With SGLang generation, adapters preload and stay pinned by default; choose on-demand loading to load them on first use and evict unpinned ones as slots fill. Each request selects an adapter and different adapters can share a generation batch, while embedding, reranking and extraction still need their own model families.