ReferenceAsk
Remy Reference/Architecture/Architecture/Retrieval
33Architecture

Retrieval

How a data source is ingested, chunked, embedded, and indexed in Qdrant, how the three-stage search (hybrid, rerank, adaptive) reads it, and how S3 keeps the index a rebuildable cache across a shared pool and dedicated instances.

An app's data sources are documents that are ingested, chunked, embedded, and indexed for retrieval by its methods and its agent. The index is Qdrant, run by the platform in two shapes: a shared pool for every workspace, and dedicated instances provisioned per customer. S3 is the source of truth for every data source, and any index can be rebuilt from it. This chapter covers the pipeline, the search, and the two serving shapes.


#The Pipeline

A document enters through an upload, a connector sync, or a method, and becomes a data-source job. Jobs run on two worker fleets: single documents on the ingest queue, and corpus loads as batches on the bulk queue, which has its own fleet. Each job extracts text, chunks it, contextualizes and describes images where the source has them, embeds the chunks, and writes an artifact to S3 alongside the points it upserts into the index.

Every setting that shapes a source sits on one side of a line. Pinned settings change what is written to the artifact and the index, so they are versioned on a pipeline row and migrated explicitly; changing one triggers a re-embed, and the user can see its cost. Live settings change only how stored vectors are searched, so changing one is free and retroactive. Staleness is always a comparison between a document's build and its own source's pipeline, never a global version.

Ingest runs at a bounded concurrency per pod, and the vendor capacity gate keeps a corpus load from saturating an embedding provider's key while interactive calls still need it.


Retrieval is three stages, each independently switchable, so they can be measured against one another:

  1. Hybrid. Dense embedding search fused with IDF-weighted lexical matching, so exact tokens a dense model never learned (part numbers, error codes, proper nouns) are still found.
  2. Rerank. A wide candidate set is over-fetched and a cross-encoder scores every candidate jointly with the query, keeping the best few.
  3. Adaptive. The rerank step is skipped when the first stage already produces an unambiguous winner.

An agent may search several times in one turn, so latency multiplies across every stage on that path, and each stage is tuned with that in mind. A search on a tenant that reads empty triggers a rebuild from the artifacts, so an evicted or lost index heals on first use.


#The Shared Pool

The shared Qdrant runs as a StatefulSet on its own node group with local NVMe, so a platform deploy never rolls it, and a pod restart lands on the same node. Each data source is a tenant in it. The pool is a cache: S3 is the source of truth; a tenant hydrates lazily on first read, serves while resident, and is evicted under disk pressure by an application-level LRU that runs well before Kubernetes would evict the pod. Eviction is deletion; rehydration is a rebuild from the artifacts; a small tenant rebuilds inside the search that found it cold, and a large one hands off to a checkpointed copy machine while the search reports progress.

The optimizer runs during loads, so appendable segments become immutable ones, and the sparse and full-text indexes honor their on-disk settings. A search against a still-indexing collection answers with progress rather than brute-forcing unindexed points.


#Dedicated Retrieval

A customer can lease a dedicated Qdrant instance. The platform's provisioner reconciles a desired state in Postgres against the cluster: it creates a StatefulSet and headless Service per resource in a dedicated namespace, applies a NetworkPolicy admitting only the platform's namespace, and drives the resource through its lifecycle phases (active, hibernating, hibernated) with idempotent, level-triggered passes and timeouts, so a restart mid-transition changes nothing.

The dedicated pods hold no AWS identity and talk to nothing but their clients. Their store is a cache of the customer's own artifacts in S3, rebuilt by the platform's workers whenever a pod comes up empty. Offerings are code-defined products with a spec, a cost basis derived from dated list prices, and a margin, so a reader can see how a price is composed. A resource freezes its offering's spec at provision time and is billed on a schedule.

What This Gives an App
Retrieval that finds exact identifiers as well as meaning, reranked for precision.
Ingest that never blocks interactive work and re-embeds only what a change actually invalidates.
An index that is disposable and rebuildable, never a single point of data loss.
The option of a dedicated instance, isolated by namespace and network policy, on the same pipeline.
DiagramFIG. 33 — RETRIEVAL
A sourceupload · connector · methodbecomes a jobIngest workerextract · chunk · describe imagesembed the chunksThe index · Qdrantpoints upserted herea cache: shared or dedicatedS3the artifact per documentthe source of truthHybrid searchdense + lexical (IDF)Rerankcross-encoder over candidatesAdaptiveskip when unambiguousResults to the method or agentRETRIEVAL · INGEST, INDEX, SEARCHpointsartifactrebuild from artifactssettings: pinned (re-embed) vs live (free)shared pool, or dedicated (own namespace)
ingest · index · search stepwrite · rebuild · search flow
Ingested once, searchable two ways, rebuildable always. A source (an upload, a connector sync, or a method) becomes a job on the ingest fleet, and the worker extracts text, chunks it, describes any images, and embeds the chunks. It writes two things: the artifact to S3, which is the source of truth, and the points to the index, which is Qdrant. The index is a cache: any tenant rebuilds from its artifacts on a cold read or after eviction, so an evicted or lost index heals on first use and is never a single point of data loss. The index comes in two shapes, a shared pool with a tenant per source, and a dedicated instance per customer in its own namespace behind a network policy. Search reads the index in three stages, each independently switchable: hybrid search fuses dense embeddings with lexical matching so exact identifiers and meaning are both found, a cross-encoder reranks the candidates, and an adaptive stage skips the rerank when the first stage already has an unambiguous winner. Settings split in two: pinned settings change what is written and cost a re-embed, while live settings change only how stored vectors are searched and are free and retroactive. No trust boundary is drawn here, so there is no crimson.

``` Figure "Retrieval": S3 is the source of truth, the index is a rebuildable cache. Left, ingest: a source (upload, connector sync, or method) becomes a data-source job. Single documents go to the ingest queue; batches go to the bulk queue, which has its own fleet. A worker extracts text, chunks it, contextualizes it, and embeds the chunks. It writes two things: the artifact to S3 (the source of truth) and the points upserted into the index. A badge on the pipeline row reads "pinned vs live settings". Center, the index in two shapes: Shared Qdrant (a StatefulSet on its own node group with NVMe): a tenant per source, lazy hydration, LRU eviction, and a rebuild from the artifacts on a cold read. Dedicated Qdrant (one per customer resource, in a dedicated namespace, with a NetworkPolicy admitting only the platform): the provisioner reconciles a desired state in Postgres into a StatefulSet and Service, with lifecycle phases and timeouts. It is rebuilt from the customer's own artifacts. Right, search: hybrid (dense and lexical, fused by reciprocal rank fusion), then rerank (a cross-encoder over the candidates), then adaptive (skip the rerank when the first stage is unambiguous), then results to the method or the agent's search tool.