Start a bulk job
Starts a bulk ingestion job from a file store or a manifest, estimating its cost before it runs.
admin.dataSourceJobs.start({ slug, store?, prefix?, manifest?, limit?, budgetDollars?, approve?, wait? })
Loads a large set of objects into a corpus as one job, reading either from a file store (with a prefix) or from a JSONL manifest. The job first plans: it samples the set, projects the documents, chunks, tokens, and cost, and pauses as planned for approval when the projection is over the budget. --approve starts it right away, and --wait blocks to the end.
Parameters
--source
string
Corpus slug to load into.
--store
name
A file store to read objects from (with optional --prefix and --access).
--manifest
file.jsonl
A JSONL manifest of objects to load, instead of a store.
--budget
dollars
Ceiling the job auto-approves under; over it, it waits for approve.
--approve
flag
Approve the plan immediately, subject to the capacity and budget gates.
--wait
flag
Block until the job finishes.
CLI
remy-admin datasources jobs start --source docs --store uploads --prefix batch/ --budget 20 --wait