Quality Evaluation

Quality evaluation answers one question: given your queries, does the model surface the right documents? You need a corpus, queries, and relevance judgments. Rank metrics such as NDCG@10 do the rest. The SIE repository ships a complete example of exactly this workflow.

The Shipped Benchmark

examples/retrieval-ablation benchmarks page-level document search on six bank 10-K filings from SEC EDGAR. The script does four things:

1. Encodes 2,942 pages with dense and multi-vector models through sie.encode.
2. Encodes 1,854 real queries the same way.
3. Runs six ablation conditions: BM25, dense vector search, RRF fusion, cross-encoder reranking, multi-vector reranking, and multi-vector direct search.
4. Scores every ranked list with NDCG@10, MRR@10, and Recall@10 against 8,766 relevance judgments.

Eight models across the sweep (a dense encoder, three cross-encoder rerankers, and five multi-vector models, with BAAI/bge-m3 pulling double duty as encoder and multi-vector model), one SIE endpoint, no model serving to manage. The docs write-up covers the methodology and results in full.

Run It Against Your Endpoint

Create a .env in the example directory first. The example also uses Turbopuffer for its BM25 and vector index:

# Use https for any remote endpoint; reserve http:// for local,
# unauthenticated development, since SIE_API_KEY travels as a bearer token.
SIE_BASE_URL=https://your-sie-endpoint
# Optional: only needed for managed/auth-enabled SIE clusters.
SIE_API_KEY=
TURBOPUFFER_API_KEY=tpuf_...

Then:

uv sync

# Validate config (no GPU needed)
uv run python benchmark_ablation.py --dry-run

# Full run: every condition and model sweep, all 1,854 queries
uv run python benchmark_ablation.py --gpu l4-spot

# Skip the baselines (BM25, dense, RRF); still sweeps every reranker
# and multi-vector model in conditions 4-6
uv run python benchmark_ablation.py --gpu l4-spot --skip-conditions 1,2,3

Encoding and search results cache to cache/ablation/, so re-runs skip completed steps. Cross-encoder reranking checkpoints every 100 queries.

What It Found

All figures below come from the example’s RESULTS.md:

Strategy	NDCG@10	Recall@10
Dual multi-vector pool, then mxbai-rerank-large-v2	0.621	0.665
Cross-encoder rerank over a hybrid BM25+vector pool	0.600	0.640
bge-m3 multi-vector direct	0.435	0.482
bge-m3 dense vector	0.396	0.438
BM25	0.185	0.239

Reranking dominated. RRF fusion actually scored below plain vector search on this dataset (0.358 vs 0.396): BM25 diluted a strong vector signal. That is the point of running evals on your own data; the result was not obvious in advance.

Running Your Own MTEB-Style Eval

Any MTEB or BEIR retrieval task decomposes into the same three parts: a corpus, queries, and qrels. The loop is short:

1. Encode the corpus with client.encode (batch the calls).
2. Encode queries with is_query=True.
3. Rank by similarity, and optionally rerank the top candidates with client.score.
4. Compute NDCG@10 against the qrels. The shipped benchmark implements ndcg_at_k, mrr_at_k, and recall_at_k in plain Python; nothing heavier is required.

Custom Evals walks through this pattern with runnable code.

What’s Next

• Custom Evals - the worked pattern on your own labeled data
• Performance Evaluation - latency and throughput measurement
• Reranking models - choosing a cross-encoder to test