?
Create an eval set
Builds an evaluation set for a corpus: a batch of queries, each tagged with the document it should return.
admin.dataSourceEvals.createSet({ slug, name, size?, style?, model?, tags?, description?, seed?, wait? })
Generates an evaluation set from a corpus by sampling passages and writing a query for each, tagged with the document that answers it. --style question writes natural questions; cloze writes fill-in-the-blank probes. The set is the fixed yardstick a run scores against.
Parameters
--source
string
RequiredCorpus to build the set from.
--name
string
RequiredName for the eval set.
--size
number
How many queries to generate.
--style
cloze|question
Fill-in-the-blank probes, or natural questions.
--model
id
Model used to write the queries.
CLI
remy-admin datasources eval create --source news --name baseline --size 100 --style question