?
Add an eval query
Adds one query to a set, naming the document it should return.
admin.dataSourceEvals.addQueries(setId, [{ query, expect }])
Adds a hand-written query to a set, alongside the generated ones, naming the document (by id, filename, or external id) that a good retriever should return for it. Use it to pin the questions you actually care about into the yardstick.
Parameters
--set
id
RequiredEval set to add to.
--query
text
RequiredThe query text.
--expect
ref,...
RequiredThe document(s) that should come back: documentId, filename, or externalId.
--passage
text
The passage the query is drawn from, for context.
CLI
remy-admin datasources eval add --set set_abc123 --query 'what is the refund window?' --expect refunds.pdf