DataSourceJob

Type

One bulk ingestion job: its state, cost plan, progress counts, quarantine, and why it last paused.

One bulk ingestion job in full. state runs planning to planned to running (pausing and resuming) to done, failed, or cancelled; plan is the cost projection it paused on; counts track documents done and failed and chunks embedded; quarantine holds what a mapper set aside; and pauseReason says why a running job stopped. progress carries the live rate and ETA.

DataSourceJob.ts

interface DataSourceJob {
  id: string
  dataSourceId: string
  source: DataSourceJobSource
  state: DataSourceJobState
  plan: DataSourceJobPlan | null
  budgetDollars: number | null
  counts: { documentsDone: number; documentsFailed: number; chunks: number }
  estimatedDollars: number
  quarantine: { skipped: number; errors: number }
  pauseReason: 'user' | 'budget' | 'errors' | 'skips' | null
  progress: OperationProgress
  createdAt: string
  finishedAt: string | null
}

Fields

state

DataSourceJobState

planning, planned, running, paused, done, failed, or cancelled.

source

DataSourceJobSource

Where the job reads from: a store, a manifest, a connector, a sample, a remap, or a replay.

plan

DataSourceJobPlan | null

The cost projection a planned job waits on for approval.

counts

object

Documents done and failed, and chunks embedded, so far.

quarantine

{ skipped, errors }

Objects a mapper set aside, by kind.

pauseReason

'user'|'budget'|'errors'|'skips'|null

Why a running job stopped.

progress

OperationProgress

The live rate and ETA against the plan's projection.

Wait on a job

const { job } = await admin.dataSourceJobs.get({ id });
if (job.state === 'planned') await admin.dataSourceJobs.approve({ id });