Retrieval

An app's data sources are documents that are ingested, chunked, embedded, and indexed for retrieval by its methods and its agent. The index is Qdrant, run by the platform in two shapes: a shared pool for every workspace, and dedicated instances provisioned per customer. S3 is the source of truth for every data source, and any index can be rebuilt from it. This chapter covers the pipeline, the search, and the two serving shapes.

#The Pipeline

A document enters through an upload, a connector sync, or a method, and becomes a data-source job. Jobs run on two worker fleets: single documents on the ingest queue, and corpus loads as batches on the bulk queue, which has its own fleet. Each job extracts text, chunks it, contextualizes and describes images where the source has them, embeds the chunks, and writes an artifact to S3 alongside the points it upserts into the index.

Every setting that shapes a source sits on one side of a line. Pinned settings change what is written to the artifact and the index, so they are versioned on a pipeline row and migrated explicitly; changing one triggers a re-embed, and the user can see its cost. Live settings change only how stored vectors are searched, so changing one is free and retroactive. Staleness is always a comparison between a document's build and its own source's pipeline, never a global version.

Ingest runs at a bounded concurrency per pod, and a vendor capacity gate stops a corpus load from saturating an embedding provider's key while interactive calls still need it.

#Search

Retrieval is three stages, each independently switchable, so they can be measured against one another:

1. Hybrid. Dense embedding search fused with IDF-weighted lexical matching, so exact tokens a dense model never learned (part numbers, error codes, proper nouns) are still found.
2. Rerank. A wide candidate set is over-fetched and a cross-encoder scores every candidate jointly with the query, keeping the best few.
3. Adaptive. The rerank step is skipped when the first stage already produces an unambiguous winner.

An agent may search several times in one turn, so latency multiplies across every stage on that path, and each stage is tuned with that in mind. A search on a tenant that reads empty triggers a rebuild from the artifacts, so an evicted or lost index heals on first use.

#The Shared Pool

The shared Qdrant runs as a StatefulSet on its own node group with local NVMe, so a platform deploy never rolls it, and a pod restart lands on the same node. Each data source is a tenant in it. The pool is a cache: S3 is the source of truth; a tenant hydrates lazily on first read, serves while resident, and is evicted under disk pressure by an application-level LRU that runs well before Kubernetes would evict the pod. Eviction is deletion; rehydration is a rebuild from the artifacts; a small tenant rebuilds inside the search that found it cold, and a large one hands off to a checkpointed copy machine while the search reports progress.

The optimizer runs during loads, so appendable segments become immutable ones, and the sparse and full-text indexes honor their on-disk settings. A search against a still-indexing collection answers with progress rather than brute-forcing unindexed points.

#Dedicated Retrieval

A customer can lease a dedicated Qdrant instance. The platform's provisioner reconciles a desired state in Postgres against the cluster: it creates a StatefulSet and headless Service per resource in a dedicated namespace, applies a NetworkPolicy admitting only the platform's namespace, and drives the resource through its lifecycle phases (active, hibernating, hibernated) with idempotent, level-triggered passes and timeouts, so a restart mid-transition changes nothing.

The dedicated pods hold no AWS identity and talk to nothing but their clients. Their store is a cache of the customer's own artifacts in S3, rebuilt by the platform's workers whenever a pod comes up empty. Offerings are code-defined products with a spec, a cost basis derived from dated list prices, and a margin, so a reader can see how a price is composed. A resource freezes its offering's spec at provision time and is billed on a schedule.