Monitoring & Observability

SIE exposes monitoring across the control plane and worker pods. Inside each Kubernetes worker pod, the SIE server sidecar owns queue health; the Python sie-server adapter owns model execution. Use health endpoints for orchestration. Application metrics leave each process once over OTLP, then the bundled OpenTelemetry Collector routes them to Prometheus and Grafana, with optional fan-out to a remote observability backend. WebSocket streams provide interactive status.

Health Endpoints

SIE exposes Kubernetes-compatible health probes for liveness and readiness checks. In Docker, the Python sie-server process owns these endpoints. In Kubernetes, the gateway, config service, and both containers inside each worker pod have their own health contract.

Component	`/healthz`	`/readyz`
`sie-gateway`	Process liveness, returns `ok`	Process readiness. It does not wait for SIE server sidecar health or `sie-config`
SIE server sidecar (`worker-sidecar` container)	Process liveness	Fresh IPC `Ping` to the in-pod Python process and no active drain
`sie-server`	Python process liveness	Adapter process ready to receive work
`sie-config`	Config process liveness	Registry initialized and able to serve config endpoints

Liveness

curl http://localhost:8080/healthz
# Returns: ok

Use /healthz for Kubernetes liveness probes. A failed check triggers container restart.

Readiness

curl http://localhost:8080/readyz
# Returns: ok

Use /readyz for Kubernetes readiness probes. On the gateway, readiness means the process can accept traffic and return 503 PROVISIONING for cold-start capacity; worker-pod availability is exposed through /health, inference responses, and metrics.

Kubernetes configuration:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

OpenTelemetry Metrics

SIE services emit canonical OpenTelemetry metrics once over OTLP. Application containers do not expose a Prometheus /metrics endpoint. The bundled OpenTelemetry Collector receives OTLP and exposes Prometheus-compatible metrics on port 9464. It can also fan out the same observations to a remote OTLP backend.

Canonical OpenTelemetry names use dots. The collector owns the underscore-form Prometheus compatibility names used by PromQL, Grafana, alerts, and KEDA. The checked-in telemetry contract defines the full inventory, attributes, histogram boundaries, and export eligibility.

Metrics Reference

Canonical OpenTelemetry name	Prometheus wire name	Type	Description
`sie.gateway.requests`	`sie_gateway_requests_total`	Counter	Gateway inference responses by operation, outcome, HTTP status, and machine profile
`sie.gateway.request.duration`	`sie_gateway_request_duration_seconds`	Histogram	End-to-end gateway request latency
`sie.gateway.lane.queue.depth`	`sie_gateway_lane_queue_depth`	Gauge	Exact JetStream pending and delivered-but-unacknowledged work by physical lane
`sie.gateway.lane.queue.snapshot.timestamp`	`sie_gateway_lane_queue_snapshot_timestamp_seconds`	Gauge	Freshness companion for exact-lane queue depth
`sie.worker.requests`	`sie_worker_requests_total`	Counter	Completed worker items by operation, outcome, backend, lane, model, and profile
`sie.worker.request.duration`	`sie_worker_request_duration_seconds`	Histogram	Per-item worker latency
`sie.worker.batch.size`	`sie_worker_batch_size`	Histogram	Items in each formed batch
`sie.worker.queue.depth`	`sie_worker_queue_depth`	Gauge	Current worker queue depth
`sie.worker.model.loaded`	`sie_worker_model_loaded`	Gauge	Current model residency
`sie.worker.model.memory`	`sie_worker_model_memory_bytes`	Gauge	Model memory reported by the active backend

Enable Prometheus and Grafana

The bundled kube-prometheus-stack installs Prometheus, Grafana, Alertmanager, and the Prometheus Operator. Enabling it also creates the collector, its ServiceMonitor, the SIE alert rules, and dashboard ConfigMaps:

helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  --version 0.6.26 \
  --namespace sie \
  --create-namespace \
  --set kube-prometheus-stack.install=true

With an existing Prometheus Operator, set serviceMonitor.enabled=true instead. If Prometheus runs outside the SIE namespace, add its namespace to observability.otel.collector.prometheus.networkPolicy.scrapeNamespaceNames.

Manual Prometheus Scrape

Scrape the collector, not the gateway or workers. For a release named sie in namespace sie, the static equivalent of the chart-managed ServiceMonitor is:

# prometheus.yml
scrape_configs:
  - job_name: 'sie-otel-collector'
    static_configs:
      - targets: ['sie-sie-cluster-otel-collector.sie.svc:9464']
    metrics_path: /metrics
    scrape_interval: 5s

Live Status

For live status without the Grafana stack, SIE has two built-in surfaces:

WebSocket stream: the Python sie-server process streams real-time server, GPU, and model status over /ws/status (see WebSocket Status below).
Gateway health polling: poll the gateway /health endpoint for aggregate cluster status (worker count, GPUs, loaded models).

WebSocket Status

The Python sie-server process streams real-time status over WebSocket at /ws/status. Updates push every 200ms. In Kubernetes, the gateway also exposes /ws/cluster-status for aggregate cluster status, while routing health comes from SIE server sidecar NATS heartbeats.

Connection

import asyncio
import websockets
import json

async def monitor():
    async with websockets.connect("ws://localhost:8080/ws/status") as ws:
        async for message in ws:
            status = json.loads(message)
            print(f"Loaded models: {status['loaded_models']}")
            print(f"GPU type: {status['gpu']}")

Status Message Format

{
  "timestamp": 1703001234.567,
  "gpu": "l4",
  "loaded_models": ["bge-m3", "e5-base-v2"],
  "server": {
    "version": "0.1.0",
    "uptime_seconds": 3600,
    "user": "sie",
    "working_dir": "/app",
    "pid": 1
  },
  "gpus": [
    {
      "device": "cuda:0",
      "name": "NVIDIA L4",
      "gpu_type": "l4",
      "utilization_pct": 45,
      "memory_used_bytes": 8589934592,
      "memory_total_bytes": 23622320128,
      "memory_threshold_pct": 95
    }
  ],
  "models": [
    {
      "name": "bge-m3",
      "state": "loaded",
      "device": "cuda:0",
      "memory_bytes": 2147483648,
      "queue_depth": 0,
      "queue_pending_items": 0,
      "config": {
        "hf_id": "BAAI/bge-m3",
        "adapter": "bge_m3",
        "inputs": ["text"],
        "outputs": ["dense", "sparse"]
      }
    }
  ],
  "counters": {},
  "histograms": {}
}

Model States

State	Description
`available`	Config loaded, weights not in memory
`loading`	Weights currently loading to GPU
`loaded`	Ready for inference
`unloading`	Weights being evicted from GPU
`failed`	Last load attempt failed; config still present

Collector-backed Grafana Dashboards

SIE includes pre-built Grafana dashboards in the Helm chart at deploy/helm/sie-cluster/files/dashboards/. Grafana’s sidecar provisions them automatically. The collector can route the same OTLP observations to a remote backend without changing application instrumentation.

These example PromQL queries use the collector’s compatibility names and pin a release named sie in namespace sie. Change namespace and collector service for your release; keep endpoint="prometheus".

Request Rate

sum by (operation) (
  rate(sie_gateway_requests_total{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-gateway",
    outcome="success"
  }[5m])
)

P99 Latency

histogram_quantile(0.99,
  sum by (le, operation) (
    rate(sie_gateway_request_duration_seconds_bucket{
      namespace="sie",
      service="sie-sie-cluster-otel-collector",
      endpoint="prometheus",
      producer_service="sie-gateway"
    }[5m])
  )
)

GPU Memory Usage

max by (model, profile, backend, lane) (
  sie_worker_model_memory_bytes{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-worker"
  }
)

Queue Depth

max by (pool, machine_profile, bundle) (
  sie_gateway_lane_queue_depth{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-gateway"
  }
  and on (producer_instance, collector_generation, pool, machine_profile, bundle)
  (
    abs(time() - sie_gateway_lane_queue_snapshot_timestamp_seconds{
      namespace="sie",
      service="sie-sie-cluster-otel-collector",
      endpoint="prometheus",
      producer_service="sie-gateway"
    }) < 20
  )
)

Batch Efficiency

avg by (model, profile) (
  sie_worker_batch_fill_ratio{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service=~"sie-worker|sie-worker-sidecar"
  }
)

Alert Rules

The sie-cluster chart can render pre-configured Prometheus alert rules:

Alert	Severity	Condition	Description
`SIEWorkerDown`	critical	SIE server sidecar container not ready for 2 min	A worker pod is unavailable
`SIENoHealthyWorkers`	critical	No ready SIE server sidecar containers for 1 min	All worker pods are unavailable
`SIEWorkerHighQueueDepth`	warning	Fresh lane queue depth > 50 for 5 min	The physical lane may need more capacity
`SIEGPUMemoryHigh`	warning	GPU memory > 90% for 5 min	Risk of OOM, LRU eviction may be insufficient
`SIEGPUTemperatureHigh`	warning	GPU temp > 80°C for 5 min	GPU throttling likely, check cooling
`SIEGPUECCErrors`	critical	Double-bit ECC errors increase over 1h	Hardware issue likely
`SIEGatewayDown`	critical	No ready gateway containers for 1 min	Traffic cannot be routed
`SIEHighErrorRate`	warning	Gateway 5xx rate > 5% for 5 min	Server or model errors spiking
`SIEHighLatency`	warning	p95 latency > 5s for 5 min	Request latency is above target
`SIEGenerationServerErrorSpike`	warning	Generation 5xx rate > 0.02 req/s for 5 min	Generation failures are increasing on a machine profile
`SIEModelLoadingRetrySpike`	info	Model-load errors or timeouts > 0.05/s for 10 min	Workers are repeatedly failing to load a model
`SIEResourceExhaustedSpike`	info	Terminal OOM recovery > 0.1/s for 10 min	Worker capacity is constrained
`SIEConfigDown`	critical	Config container not ready for 2 min	Config writes are blocked; gateways serve cached state
`SIEProvisioningStuck`	warning	Pod Pending for 10 min	Check scheduling events and GPU capacity
`SIEScaleUpFailed`	warning	FailedScheduling event in 10 min	Likely insufficient GPU capacity

Installing Alert Rules

The bundled kube-prometheus-stack command above installs the alert rules automatically. With an existing Prometheus Operator, enable them explicitly:

helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  --version 0.6.26 \
  --namespace sie \
  -f helm-values.yaml \
  --set alertRules.enabled=true

Custom Alerts

Add custom alerts to your Prometheus configuration:

# Alert when P99 latency exceeds 5 seconds
- alert: SIEHighLatencyP99
  expr: |
    histogram_quantile(0.99,
      sum by (le, operation) (
        rate(sie_gateway_request_duration_seconds_bucket{
          namespace="sie",
          service="sie-sie-cluster-otel-collector",
          endpoint="prometheus",
          producer_service="sie-gateway"
        }[5m])
      )
    ) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High P99 latency for {{ $labels.operation }}"

Logging

SIE supports both human-readable and structured JSON logging.

Log Levels

Enable verbose logging with --verbose or -v:

sie-server serve --verbose

JSON Logging

Enable JSON format for Loki and log aggregation systems:

sie-server serve --json-logs

Or via environment variable:

export SIE_LOG_JSON=true
sie-server serve

JSON Log Format

{
  "timestamp": "2025-12-18T10:30:00.123Z",
  "level": "INFO",
  "logger": "sie_server.api.encode",
  "message": "Inference completed",
  "model": "bge-m3",
  "request_id": "abc123",
  "trace_id": "def456",
  "latency_ms": 45.2,
  "batch_size": 16,
  "gpu_type": "l4"
}

Structured Fields

JSON logs include optional fields when available:

Field	Description
`model`	Model name for the request
`request_id`	Unique request identifier
`trace_id`	OpenTelemetry trace ID
`latency_ms`	Request latency in milliseconds
`batch_size`	Number of items in the batch
`gpu_type`	Detected GPU type

What’s Next

Scale-from-Zero - autoscaling lifecycle and troubleshooting
Troubleshooting - common issues and solutions
CLI Reference for all server options
API Reference for endpoint documentation