?
Run an eval
Runs a set against a corpus at a chosen retrieval config, scoring recall, MRR, and NDCG.
admin.dataSourceEvals.run({ setId, source?, candidate?, label?, mode?, rerank?, topK?, wait? })
Runs every query in the set against the corpus and scores the results: recall at 1, 5, and 10, MRR, NDCG, and latency, broken down by tag and by which retrieval branch found each hit. The retrieval knobs (--mode, --rerank, --top-k) are the thing under test, so a run pins a config and reports what it scores.
Parameters
--set
id
RequiredEval set to run.
--source
string
Corpus to run against; defaults to the set's own.
--candidate
flag
Run against a rebuild in progress rather than what is live.
--mode
hybrid|semantic|lexical
Retrieval mode under test.
--rerank
true|false
Whether reranking is on for this run.
--top-k
number
How many results to score against.
CLI
remy-admin datasources eval run --set set_abc123 --mode hybrid --rerank true --wait