EvalRun

Type

One evaluation run: a set scored against a corpus at a fixed retrieval config, with recall, MRR, and NDCG.

One evaluation run: a set scored against a corpus at a fixed retrieval config (retrieval). aggregates carries the scores once it finishes: recall at 1, 5, and 10, MRR, NDCG, latency, a per-tag breakdown, and which retrieval branch found each hit. compare diffs two of these.

EvalRun.ts

interface EvalRun {
  id: string
  setId: string
  dataSourceId: string
  retrieval: EvalRetrieval
  state: EvalRunState
  completed: number
  total: number
  aggregates: EvalAggregates | null
  estimatedDollars: number
}

interface EvalAggregates {
  recall: { at1: number; at5: number; at10: number }
  mrr: number
  ndcg10: number
  latency: { p50: number; p95: number; mean: number }
  branches: { denseOnly: number; lexicalOnly: number; both: number }
  byTag: Record<string, { recallAt5: number; mrr: number }>
}

Fields

retrieval

EvalRetrieval

The config under test: mode, rerank, top-k.

aggregates

EvalAggregates | null

The scores once the run finishes; null while it is still running.

aggregates.recall

{ at1, at5, at10 }

Share of queries whose expected document came back in the top 1, 5, and 10.

aggregates.branches

object

Which retrieval branch found the expected document: dense, lexical, or both.

Read a run's scores

const { run } = await admin.dataSourceEvals.getRun(runId);
console.log(run.aggregates?.recall.at5, run.aggregates?.mrr);