Architecture

SIE has two runtime shapes. A single Docker sie-server exposes the server API in one process. A Kubernetes cluster splits the path across a Rust gateway, NATS, GPU worker pods, and a dedicated config service. Each worker pod runs the SIE server sidecar beside the Python sie-server adapter process.

System Overview

SIE system architecture: Client, Gateway, NATS, worker pod with SIE server sidecar, and sie-server adapter layers

In production Kubernetes deployments, the hot path is intentionally separate from the config control plane:

Client SDK
  -> sie-gateway (Rust, stateless inference edge)
  -> NATS JetStream queue
  -> SIE server sidecar inside the worker pod
  -> UDS IPC
  -> sie-server adapter process
  -> NATS Core result inbox
  -> sie-gateway response

Admin tooling
  -> sie-config (Python, single-writer config control plane)
  -> config store + NATS config deltas
  -> gateways and worker pods converge asynchronously

Components

Client SDK

The SDK provides encode(), score(), and extract() methods. It handles:

msgpack serialization: Binary wire format, faster and smaller than JSON
Automatic 202 retry: Waits for scale-from-zero with wait_for_capacity=True
Pool management: Background lease renewal for resource pools
Numpy integration: Returns native numpy arrays for embeddings

Framework integrations (LangChain, LlamaIndex, etc.) wrap the SDK with framework-specific interfaces.

Gateway

The gateway is a stateless Rust service that sits between clients and Kubernetes worker pods. It is optional for single-server setups but required for elastic Kubernetes clusters.

Responsibilities:

Resolves model, bundle, machine profile, and pool from its in-memory registry
Publishes inference work to NATS JetStream
Returns 202 Accepted with Retry-After when the target worker pool is scaled to zero
Serves read-side config endpoints from its local registry mirror
Manages resource pools for capacity isolation
Tracks worker-pod health and bundle config hashes from SIE server sidecar NATS heartbeats

The gateway does not own config writes. POST /v1/configs/models, GET /v1/configs/export, and GET /v1/configs/epoch belong to sie-config.

Config Service

sie-config is the authoritative config control plane. It runs as a single writer, persists API-added model configs, and publishes runtime config deltas:

POST /v1/configs/models appends new models or profiles.
GET /v1/configs/export gives gateways a full snapshot for bootstrap and drift recovery.
GET /v1/configs/epoch exposes the authoritative model-write epoch and bundle-set hash.
GET /v1/configs/bundles{,/{id}} lets gateways fetch the bundle set baked into the sie-config image.

Gateways bootstrap from sie-config, subscribe to sie.config.models._all for live deltas, and poll /v1/configs/epoch to recover any missed NATS messages.

SIE Server Sidecar

Every queue-mode worker pod includes the SIE server sidecar. The Helm chart enables it by default and renders the container as worker-sidecar. The image repository is ghcr.io/superlinked/sie-server-sidecar. It owns the queue half of the worker pod’s cluster hot path:

Pulls work from the pool’s NATS JetStream stream
Validates subjects and reply inboxes before processing payloads
Forms batches by model, operation, and LoRA key
Calls the Python sie-server adapter process over Unix domain socket IPC
Frames results, publishes them to the gateway inbox, and ACKs or NAKs JetStream messages
Publishes sie.health.<worker_id> heartbeats with queue depth, loaded model state, and the current bundle_config_hash
Applies bundle-scoped config deltas to the sie-server adapter through IPC and reconciles missed deltas from sie-config

The SIE server sidecar does not load model weights and does not link GPU libraries. It keeps queue, batching, payload, and framing work in Rust while the sie-server adapter remains responsible for model execution.

Worker Adapter (sie-server)

sie-server remains the Python model-execution process. In standalone Docker it exposes the server API. In Kubernetes queue mode, it runs beside the SIE server sidecar inside each worker pod and exposes an IPC server to execute prepared batches.

It owns:

Model registry and GPU lifecycle
Adapter-specific preprocessing and model-path selection
Model loading, LoRA loading, and memory pressure eviction
GPU inference through PyTorch, Flash Attention, SGLang, and other adapter backends
The single-process API used by Docker deployments

Queue-mode batches from the SIE server sidecar arrive as fully formed GPU work. The sie-server adapter can still retokenize an item when that is the safe execution path.

Wire Protocol

SIE uses msgpack as the default wire format instead of JSON:

Format	Encode speed	Decode speed	Size	Numpy support
msgpack	Fast	Fast	~50% of JSON	Native via msgpack-numpy
JSON	Slower	Slower	Baseline	Requires list conversion

The SDK sends and receives msgpack automatically. The OpenAI-compatible /v1/embeddings endpoint uses JSON for compatibility.

Inside a Kubernetes cluster, gateway-to-sidecar work items, IPC frames between the SIE server sidecar and the sie-server adapter, and sidecar-to-gateway results are msgpack as well. JSON is reserved for low-frequency control-plane APIs and client requests that explicitly negotiate JSON.

Model Cache Hierarchy

Model weights are resolved through a 3-tier cache:

Local disk cache uses LRU eviction when disk usage exceeds SIE_DISK_PRESSURE_THRESHOLD_PERCENT (default: 85%).

Cluster cache is useful for Kubernetes deployments where multiple worker pods share the same S3/GCS bucket, avoiding redundant downloads from HuggingFace.

Deployment Modes

Standalone Docker

Client → sie-server (single GPU)

Simplest setup. The SDK points at one sie-server process. Good for development and small production.

Multi-Bundle (Docker Compose)

Client → sie-server:8080 (default bundle)
Client → sie-server:8081 (sglang bundle)

Multiple containers, each with a different bundle. Client routes to the correct port.

Cluster (Kubernetes)

Client → sie-gateway → NATS JetStream → SIE server sidecar → UDS IPC → sie-server adapter
Admin → sie-config → config store + NATS config deltas → gateway + worker pod

Full production setup with GPU routing, autoscaling, and observability. See Kubernetes in GCP or AWS.

What’s Next

Request Pipeline - detailed preprocessing, batching, and GPU inference flow
Gateway - routing, queueing, load balancing, and resource pools
Config API - runtime model config writes and readiness polling
Adapters - compute engine abstraction layer