How to Use DSPy for Internal Tools: Compiled Prompts, Not Hand-Tuned Ones

You use DSPy for internal tools by treating your prompt as a compiled program, not a string. Define a typed signature for the task, write a metric that scores success, pick an optimizer, and let it search for the instructions and examples that actually work. Swap the underlying model later and recompile instead of rewriting prompts by hand.

That's the whole idea. The rest of this piece is implementation detail, plus why Claude Opus 5.5 has made experimenting this way cheap enough to do properly.

Why does 'just prompt it' break down at scale?

Most internal AI tools start the same way. Someone writes a prompt, tests it on five examples, ships it. It works fine until the model gets upgraded, the input distribution shifts, or someone edits one line and quality quietly drops. Nobody notices until a downstream report is wrong.

Meanwhile the SaaS answer to the same problem is to buy another AI feature bolted onto an existing tool, or a new point solution entirely. Enterprises already run an average of 118 SaaS apps, up from 106 the year before, an 11% year-over-year climb, and roughly half those licenses go unused.¹ Adding an AI-flavored subscription for every workflow that could be a fifty-line internal program is how that number keeps growing. The teams that have already lived through this math on other categories of tooling know niche SaaS is losing ground to build-vs-buy defaults for exactly this reason.

The underlying problem isn't that prompts are a bad idea. It's that a hand-written prompt has no test suite, no versioning discipline, and no way to know if it still works after a model swap. DSPy exists to fix that.

What is DSPy?

DSPy, from Stanford, reframes the job as programming, not prompting.² Instead of hand-crafting a prompt string, you define three things:

• A signature. A typed description of inputs and outputs for a task, like a function signature but for a language model call.²
• A metric. A scoring function that tells the system whether an output was good, not a vibe check.
• An optimizer. An algorithm, such as BootstrapFewShot, MIPROv2, or GEPA, that searches for the instructions and few-shot examples that maximize the metric against real examples.²³

The optimizer then compiles your program: it translates the signature and metric into model-specific instructions that actually work for the model you're running.² Change models and you recompile. You don't manually rewrite the prompt and hope.

GEPA is worth naming specifically because it's the optimizer doing most of the current heavy lifting. It uses a separate reflection model to read execution traces and textual feedback on failures, then rewrites instructions accordingly. A light-budget run evaluates roughly six candidate prompts before stopping, and critically, the same base signature compiles into different optimized instructions depending on which model you deploy it against.³ That's the part that matters for internal tools: you're not maintaining one prompt per model, you're maintaining one signature and letting the optimizer handle the translation.

Adoption critics have a fair point buried in the enthusiasm: DSPy's real value is the optimization loop, and that loop needs a dataset and a defined task to run against.⁴ If you don't have either, you're not ready for this yet. Keep that in mind before step one below.

What is Imp, DSPy's port for the BEAM?

If your internal tools already run on Elixir or Erlang, you don't need to bolt on a Python service just to get DSPy's optimizer discipline. Imp ports DSPy's core concepts to the BEAM virtual machine: signatures, modules, and optimizers including MIPROv2 and GEPA.⁵⁶ The project ships an explicit "Imp for DSPy Users" guide mapping concepts across the two ecosystems and documenting where BEAM's runtime differs from Python's, and it publishes honest conformance numbers rather than claiming one-to-one parity.⁶

The pitch for Imp isn't that it beats DSPy technically. It's that teams running BEAM infrastructure get compiled-prompt discipline without a separate language runtime, a separate deploy pipeline, and a separate on-call rotation for a Python sidecar. For a lot of internal tooling teams, that operational simplicity is worth more than framework maturity.

How to build an internal tool with DSPy: the steps

Here's the actual workflow, whether you're using DSPy in Python or Imp on the BEAM.

1. Write the signature first. Define exactly what goes in and what comes out. Resist the urge to write instructions yet. A signature for a support-ticket triage tool might be: ticket text in, category and confidence score out.
2. Collect 30 to 100 labeled examples. This is the step people skip and the step that makes everything after it work. Without examples, the optimizer has nothing to search against.
3. Write a metric, not a vibe. Decide programmatically what counts as a correct output. Exact match, a scoring rubric, or a secondary LLM judge all work, but it has to be a function you can run in a loop.
4. Pick an optimizer. Start with something cheap like BootstrapFewShot for a first pass. Move to MIPROv2 or GEPA once you have a metric and dataset that justify the extra compute.²³
5. Compile. Run the optimizer against your dataset and metric. It will generate and test candidate instructions automatically, something a light GEPA budget does in around six rounds.³
6. Evaluate on held-out data. Don't trust the training-set score. Hold back examples the optimizer never saw.
7. Swap models and recompile, don't rewrite. When a cheaper or faster model becomes available, rerun the compile step against the new model rather than manually re-tuning the prompt.

That last step is the entire point of the exercise. It's also where the return on investment shows up fastest.

A real example: Dropbox Dash's relevance judge

Dropbox's Dash team built an LLM-as-judge system to score search relevance. Manual prompt tuning got them a functioning judge, but quality plateaued early, and every model swap or small prompt edit risked regressions.⁷ They brought in DSPy to turn that manual, fragile process into a repeatable optimization loop against a measurable objective.⁷ DSPy-driven optimization is now used across more than 30 prompts in Dash's stack.⁸

Shopify ran a similar experiment: they converted a single hand-written GPT-5 prompt task into a DSPy program, moved it onto a much smaller Qwen model, and optimized it with GEPA. The result was a solution roughly 75 times cheaper and about twice as reliable as the original large-model prompt.³ That's not a marginal efficiency gain. That's the difference between running a workflow on a premium model forever and running it on hardware that costs a fraction as much, with better accuracy, because the instructions were tuned for that specific model rather than borrowed from a bigger one.

Where Claude Opus 5.5 fits in

DSPy and Imp handle the automated half of prompt engineering: searching for instructions that work against a metric. But Anthropic's own guidance for Claude Opus 5.5 makes clear that model-specific, highly situational prompting still matters on top of that. The documentation recommends starting effort at medium rather than defaulting to maximum reasoning, noting that Opus 5.5 at medium effort matches or exceeds Opus 5 at high effort on coding and knowledge-work evaluations.⁹ It also gives precise instructions for unattended agent loops and notes the model resists indirect prompt injection better than any earlier Opus release.⁹

That's the layer DSPy automates rather than replaces. Your signature and metric stay stable. The optimizer's compiled instructions absorb model-specific quirks like effort calibration or injection resistance, so you're not manually re-deriving them every time Anthropic ships an update.

The economics make this cheaper to run than it used to be. Opus 5.5 launched at $4 and $20 per million input and output tokens, 20% less than Opus 5, with cache reads at $0.20 per million tokens, a 60% cut.¹⁰ Combined with output generation more than 30% faster, Anthropic puts the net cost reduction at roughly 40% for typical workloads.¹⁰ Frequent recompilation against a metric, which used to be an expensive habit, is now a cheap one. Teams comparing model options for internal workloads are already running these tradeoffs against flash-tier alternatives like GLM and Qwen for the same reason.

When does building beat buying another SaaS seat?

The honest case for doing this internally instead of renting an AI feature comes down to control and cost compounding. A DSPy or Imp pipeline is versioned in your repo, tested against a dataset you own, and portable across models as pricing shifts. A SaaS AI feature is a black box you pay per seat or per conversation, with no visibility into what's happening when the vendor swaps models underneath you. That's the same argument already playing out in agent orchestration built on open frameworks and MCP instead of per-seat agent platforms.

The pressure toward building this in-house is already there whether or not it's sanctioned. Independent surveys put shadow AI usage anywhere from about 50% to over 80% of employees, depending on methodology, and a large share say they'd keep using unapproved tools even if the company banned them outright.¹¹¹² People are already solving these problems with whatever chatbot they have open. The question for engineering leadership isn't whether employees are doing this. It's whether to formalize it into something owned and testable. Teams that make that formalization intentional end up closer to the orchestration discipline Remy applies to internal AI workflows than to a vendor's opaque feature toggle.

Governance: making shadow AI accountable

Governance is the part most teams skip until an audit forces the question. Only about 15% of organizations have implemented formal AI governance policies, despite clear evidence that usage is already widespread.¹³ Roughly 78% of employees use AI tools without IT approval, and 46% say they'd keep doing it even if explicitly prohibited.¹³ UpGuard's numbers put employee usage of unapproved tools at 81%, with 88% of security leaders reporting the same behavior among their own staff.¹²

An internal DSPy or Imp pipeline doesn't automatically fix that gap, but it changes what governance is possible. A compiled prompt lives in version control. Its metric and evaluation results are logged. You can point an auditor at the exact dataset a prompt was tuned against and the exact score it produced. None of that exists when the actual workflow is an employee pasting company data into a personal chatbot tab.

Is this the right fit for your team? A checklist

Before committing engineering time to a DSPy or Imp pipeline, check whether the workflow actually qualifies:

• You have or can build a labeled dataset. No dataset means no metric means no optimizer, full stop.⁴
• The task repeats often enough to matter. One-off analysis doesn't justify a compiled pipeline. A daily or hourly workflow does.
• You've already hit a model-swap wall. If changing models broke your prompt once, it will again, and that's the exact pain DSPy addresses.⁷
• You're already running BEAM infrastructure, in which case Imp avoids adding a Python service just for this.⁵⁶
• The current SaaS alternative charges per seat or per conversation for something a fifty-line typed program could do at a fraction of the cost, the same math driving teams to build internal tools instead of renting them.

If most of those check out, you're not choosing between a prompt and a subscription. You're choosing between renting a black box and owning a program you can test, version, and swap underneath as models get cheaper.