How to evaluate model quality and performance in SIE
The only benchmark score that matters is the one on your data. Leaderboard averages tell you how a model performs on someone else’s queries. SIE ships the pieces to run that measurement yourself: encode and score primitives in the SDK, timing metadata on encode responses, cluster dashboards, and a complete worked benchmark you can rerun end to end.
What Ships Today
Section titled “What Ships Today”| What | Where |
|---|---|
A complete retrieval benchmark: 1,854 real queries over 2,942 pages from six SEC 10-K filings, six ablation conditions ranked by NDCG@10, runnable with uv | examples/retrieval-ablation on GitHub, with a full write-up in these docs |
SDK encode and score calls for running your own eval | Quality Evaluation and Custom Evals |
| A server-side timing breakdown on every encode response | Performance Evaluation |
| Seven Grafana dashboards in the Helm chart | Monitoring |
Quality and Performance Are Separate Questions
Section titled “Quality and Performance Are Separate Questions”Quality evals answer “does this model return the right documents”: labeled queries, a ranked list, NDCG@10. Performance evals answer “how fast, and at what load”: latency percentiles and throughput. A model can win one and lose the other, and intuition is a poor guide to either. In the shipped benchmark, hybrid RRF fusion scored below plain vector search (0.358 vs 0.396 NDCG@10), and BM25 alone managed 0.185.
How Supported Models Will Be Gated at 1.0
Section titled “How Supported Models Will Be Gated at 1.0”SIE’s compatibility policy ties the model catalog to eval gates as part of the 1.0 criteria: quality targets that exist and are met for all supported models, performance baselines documented for representative hardware (L4, A100-40GB, A100-80GB), and a full-catalog matrix evaluation passing at better than 95% on GKE. These 1.0 criteria are declared but not yet met.
The harness that runs those checks is internal, used in development only, and not published to PyPI. It is not part of the public API, so do not build against it. For your own evals, use the SDK patterns on the pages below.
Where To Go Next
Section titled “Where To Go Next”- Quality Evaluation: run the shipped benchmark against your endpoint and adapt its workflow to MTEB-style tasks.
- Performance Evaluation: timing metadata, isolated resource pools, and load-testing guidance.
- Custom Evals: a compact worked pattern for evaluating on your own labeled data.