Custom Evals

A few hundred labeled queries from your own domain settle model choice faster than any leaderboard. The pattern below is the same one the shipped retrieval-ablation benchmark runs at larger scale: encode, rank, optionally rerank, score.

Structure the Labeled Set

Three pieces: a corpus, queries, and relevance judgments (qrels).

corpus = [
    {"id": "doc1", "text": "Machine learning uses algorithms to learn from data."},
    {"id": "doc2", "text": "The weather forecast predicts rain tomorrow."},
]
queries = [
    {"id": "q1", "text": "What is machine learning?"},
]
qrels = {
    "q1": {"doc1": 2},  # higher grade = more relevant
}

Any graded scale works; the retrieval-ablation dataset uses 1 for relevant and 2 for highly relevant. Pull queries from real logs where you can, and have someone who knows the domain judge relevance. Unjudged documents count as not relevant.

Encode and Rank

Python

import numpy as np
from sie_sdk import SIEClient

client = SIEClient("http://your-sie-endpoint:8080")
MODEL = "BAAI/bge-m3"
CANDIDATES = 50  # rerank pool size; keep this larger than the final cutoff

doc_vecs = np.stack([r["dense"] for r in client.encode(MODEL, corpus)])
doc_vecs /= np.linalg.norm(doc_vecs, axis=1, keepdims=True)

rankings = {}
for query in queries:
    q = client.encode(MODEL, {"text": query["text"]}, is_query=True)
    q_vec = q["dense"] / np.linalg.norm(q["dense"])
    order = np.argsort(-(doc_vecs @ q_vec))
    rankings[query["id"]] = [corpus[i]["id"] for i in order[:CANDIDATES]]

TypeScript

import { SIEClient } from "@superlinked/sie-sdk";

const corpus = [
  { id: "doc1", text: "Machine learning uses algorithms to learn from data." },
  { id: "doc2", text: "The weather forecast predicts rain tomorrow." },
];
const queries = [{ id: "q1", text: "What is machine learning?" }];

const client = new SIEClient("http://your-sie-endpoint:8080");
const MODEL = "BAAI/bge-m3";
const CANDIDATES = 50;  // rerank pool size; larger than the final cutoff

function normalize(v: Float32Array): Float32Array {
  let n = 0;
  for (const x of v) n += x * x;
  n = Math.sqrt(n);
  return v.map((x) => x / n);
}

function dot(a: Float32Array, b: Float32Array): number {
  let s = 0;
  for (let i = 0; i < a.length; i++) s += a[i] * b[i];
  return s;
}

const docResults = await client.encode(MODEL, corpus);
const docVecs = docResults.map((r) => normalize(r.dense!));

const rankings: Record<string, string[]> = {};
for (const query of queries) {
  const q = await client.encode(MODEL, { text: query.text }, { isQuery: true });
  const qVec = normalize(q.dense!);
  rankings[query.id] = docVecs
    .map((v, i) => ({ id: corpus[i].id, sim: dot(v, qVec) }))
    .sort((a, b) => b.sim - a.sim)
    .slice(0, CANDIDATES)
    .map((e) => e.id);
}

encode accepts a list and returns results in the same order; chunk large corpora into batches. Query-side encoding (is_query=True in Python, isQuery: true in TypeScript) matters for models with asymmetric encoding such as BGE and E5.

Rerank the Top Candidates

Cross-encoder reranking won every condition in the shipped benchmark, and reranker size was the single biggest quality lever (+14.5% NDCG@10 for the large reranker over the base). Pool recall was the bottleneck: the same reranker scored 0.600 NDCG@10 on an 89-candidate pool and 0.621 on a 151-candidate pool (per its RESULTS.md). So retrieve more candidates than you intend to keep, then let the reranker order them:

Python

K = 10  # final cutoff for the metrics below

by_id = {c["id"]: c for c in corpus}
for query in queries:
    pool = [by_id[doc_id] for doc_id in rankings[query["id"]]]
    result = client.score(
        "mixedbread-ai/mxbai-rerank-large-v2",
        query={"text": query["text"]},
        items=pool,
    )
    rankings[query["id"]] = [s["item_id"] for s in result["scores"]][:K]

TypeScript

const K = 10;  // final cutoff for the metrics below

const byId = new Map(corpus.map((c) => [c.id, c]));
for (const query of queries) {
  const pool = rankings[query.id].map((docId) => byId.get(docId)!);
  const result = await client.score(
    "mixedbread-ai/mxbai-rerank-large-v2",
    { text: query.text },
    pool,
  );
  rankings[query.id] = result.scores.map((s) => s.itemId).slice(0, K);
}

Scores come back sorted by relevance, most relevant first, with each item’s id echoed.

Score the Ranking

NDCG and recall are a few lines of plain Python. These match the metric functions the retrieval-ablation benchmark ships:

import math

def ndcg_at_k(ranked_ids, rels, k=10):
    dcg = sum(rels.get(d, 0) / math.log2(i + 2) for i, d in enumerate(ranked_ids[:k]))
    ideal = sorted(rels.values(), reverse=True)[:k]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def recall_at_k(ranked_ids, rels, k=10):
    relevant = {d for d, r in rels.items() if r > 0}
    return len(relevant & set(ranked_ids[:k])) / len(relevant) if relevant else 0.0

ndcg = sum(ndcg_at_k(rankings[qid], qrels[qid]) for qid in qrels) / len(qrels)

To compare models, swap MODEL and rerun. Keep the corpus, queries, and qrels frozen while you iterate; a moving eval set cannot tell you whether the model improved.

For a full multi-strategy comparison template (BM25, dense, hybrid fusion, multi-vector, cross-encoder rerank), start from examples/retrieval-ablation and swap in your own dataset.

What’s Next

• Quality Evaluation - the shipped benchmark and MTEB-style workflows
• Performance Evaluation - latency and throughput measurement
• SDK Reference - full encode and score signatures