ReferenceAsk
Back to Remy
Remy Reference/Architecture/Architecture/Self-Hosted Models and Jewels
34Architecture

Self-Hosted Models and Jewels

How a jewel shadows a method and grades its proposals into a ledger, how that ledger becomes a training set, and how the platform trains and serves a per-method tuned model on its own GPUs, closing the loop back into the method.

A method can have a jewel: a shadow companion that watches how humans handle the method, proposes what it would have done, and is graded by what they actually did. The agreement ledger that accumulates becomes training data. The platform trains an adapter on it, using its own GPUs, and serves the tuned model from its own pool. The whole loop, ordinary use turning into an org-private model, runs inside the platform. This chapter covers the shadow run, the ledger, training, and serving.

#Jewels

A method declares its autonomy and a jewel in the manifest. From then on, every human invocation of that method runs the jewel beside it:

  1. Gate. The invocation is checked against the jewel's preconditions.
  2. Mirror. Before the human's method runs, the platform snapshots each of the release's databases (on its owner pod) into a local mirror, so the jewel sees the world as it was when the human began, not after their writes landed. Mirrors are local, never flushed, and disposed when the run settles. See App Databases.
  3. Run in parallel. The jewel runs against the mirror while the human's method runs against the live databases. It proposes; it does not act.
  4. Grade. When the human's run succeeds, the pair is recorded: what the jewel proposed, what the human did, and the jewel's own reasoning. A failed human run is not a demonstration and inserts no pair.

A jewel can also propose ahead of a human, on an arrival: a pending proposal that the human's eventual action grades, so there is one jewel run per decision moment. Proposals a human never acts on within their window expire as unlabeled, never as disagreements. A queue lets a reviewer approve or correct proposals directly, and an approval applies the method as the reviewer.

The pair ledger is retained long-term by design: it is the workspace's agreement evidence and, later, its training set. A dashboard shows agreement rate per jewel and lets a workspace decide when a jewel has earned more autonomy.

#From Ledger to Dataset

Training data is exported from the ledger deterministically. Agreements with captured traces become supervised examples, with the teacher's completion verbatim as the target. Corrections, whether edited approvals or graded disagreements, become preference pairs: the model's own completion is rejected and the human's decision is chosen. The train and evaluation split is a hash of the pair id, so a re-export is byte-identical and a pair never migrates between splits as the ledger grows. Everything excluded is counted, so the export doubles as a coverage report.

All of this happens on the platform at enqueue time: the dataset is written to the workspace's region in S3, a run row is inserted, and one message carries the pointers.

#Training

Training runs as a Kubernetes Job per message. KEDA spawns the job from the training queue onto the GPU node group, which scales from zero to run it.

The trainer container syncs the dataset, then trains a LoRA adapter with supervised fine-tuning. When preference pairs exist, it also runs a preference stage. It replays the held-out split for an evaluation report, uploads the adapter and the report into the app's own file store where the workspace can see them, and reports progress and completion through an internal callback. It never touches the platform database. Retry belongs to the queue: a message is redelivered on failure and parked after a bounded number of attempts.

The trainer runs under its own workload identity, with access to the training queue and the artifact prefixes, and nothing else.

#Serving

Tuned models serve from a resident pool per base model: an upstream vLLM server holding the base weights, hot-swapping every method's latest adapter onto them. Many workspaces' tuned models share one GPU, and the marginal cost of one more adapter is close to zero, which is why the menu of training bases is deliberately small.

Each pool is a Deployment on the GPU node group, running the unmodified vLLM image alongside two helpers. An init container stages the base weights and sets the chat template to the posture the adapters were trained under. A sidecar reads the platform's desired adapter set and vLLM's actual loaded set every few seconds, then converges them through the runtime adapter API. Truth comes from vLLM itself, so a restarted server self-heals to the right adapters within one poll. Rolling updates surge onto a temporary second node so a deploy is zero-downtime.

A served tuned model is an ordinary platform model. Its id names the app and the method. A call with that id goes through the model service like any other, and billing, metrics, and error handling are inherited. The pool runs under its own workload identity with read-only access to the base weights and the adapter prefixes.

What This Gives a Workspace
Automation that learns from the decisions its people already make, without a labeling project.
An agreement record it can inspect before granting a jewel any autonomy.
Training and serving that never leave the platform, on models the workspace alone can call.
A tuned model that costs what one adapter costs to load, not what a GPU costs to run.
DiagramFIG. 34 — SELF-HOSTED MODELS & JEWELS
A human runs the methodthe jewel runs beside itgate, then in parallelThe jewel shadowsreads a DB mirror, not liveproposes, does not actPair ledgerthe proposal and the outcomeagreement evidence, long-retainedExport the datasetagreements → supervised rowscorrections → preference pairsTrain the adapterLoRA: SFT + preferencea GPU Job, from zeroServe the tuned modelvLLM base + hot-swapped adaptersmany workspaces share a GPUA tuned modelan ordinary platform modelid names the app and methodSELF-HOSTED MODELS & JEWELS · THE LOOPONE LOOP, INSIDE THE PLATFORMordinary use becomes an org-private modelgraded → a pairto S3 · hash splitthe tuned model becomes the method's model
loop stagethe loop, clockwise
Ordinary use becomes an org-private model, and the whole loop runs inside the platform. A human runs a method, and the jewel runs beside it: gated, then in parallel, reading a snapshot mirror of the databases rather than live, proposing what it would have done without acting. The human’s outcome grades the proposal, and the pair, the proposal and what the human actually did, lands in a long-retained ledger that is both the workspace’s agreement evidence and its training set. From the ledger a deterministic export turns agreements into supervised examples and corrections into preference pairs, written to the workspace’s region in S3 on a hash split so a re-export is byte-identical. A GPU Job scaled from zero trains a LoRA adapter, with a supervised stage and a preference stage, and uploads the adapter and an evaluation report into the app’s own file store. A serving pool holds each base model once and hot-swaps every method’s adapter onto it, so one more tuned model costs about what one adapter costs to load. The tuned model is an ordinary platform model whose id names the app and the method, so it flows back into the method through the model service, closing the loop. No trust boundary is drawn here, so there is no crimson.

Figure 34, "Self-Hosted Models and Jewels": one clockwise loop, ordinary use becoming an org-private model. Top edge, left to right: a human runs the method; the jewel shadows it (reads a DB mirror, not live, and proposes without acting); the human's outcome grades the proposal into a long-retained pair ledger. Right edge, down: the ledger is exported into a dataset (agreements become supervised rows, corrections become preference pairs), written to S3 on a hash split; a GPU Job scaled from zero trains a LoRA adapter with a supervised stage and a preference stage. Bottom edge, right to left: a serving pool holds the base weights and hot-swaps adapters, and the result is a tuned model, an ordinary platform model whose id names the app and method. Left edge, up: the tuned model flows back into the method, closing the loop. A centred label names the loop. No trust boundary is drawn in this figure, so there is no crimson.