Patching & high tokens

Hi there, I’m pretty deep into a Remy build at this point and I’m starting to worry that I’ve gotten myself into an expensive game of whack-a-mole.

I’m nontechnical, so I came into this knowing very clearly what I wanted the product to do from the user side, but obviously not knowing how any of it needed to be architected underneath.

The first roughly $400 of the build went in the wrong direction. I didn’t realize soon enough that some crucial parts of what I’d asked for had been rewritten or changed or Remy had decided to implement things differently than I instructed. Once I understood what had happened, I had to do a lot of backtracking and rebuilding.

Since then I’ve spent roughly another $300, and to be fair, a lot has been fixed and the product is closer to what I originally intended.

My concern is that I don’t know whether we actually corrected everything at the source or whether I’m now patching around consequences of the original build.

I have an existing AI that works extremely well in its original environment, so I’ve been using that as a control to test the new system. We brought over its original instructions and knowledge and have done quite a bit of work around making that transfer faithful.

Technically, everything looks increasingly correct. Behaviorally, it still isn’t the same.

The differences aren’t obvious broken-feature things. They’re small judgment and conversational differences that add up to the AI feeling noticeably different from the original.

We’ve found at least one real underlying implementation issue that explained some of that behavior and fixed it. But since then, I’ve found myself going down a path of adjusting instructions every time another behavioral difference appears. That’s making me nervous.

I learned the hard way a while ago that continuously adding “do this / don’t do this” instructions to fix individual outputs can eventually make an AI worse. You can end up treating symptoms instead of figuring out why the behavior is different in the first place.

So I stopped doing that and ran a controlled test using the exact same AI configuration with only the underlying model changed. The behavior changed substantially (and not in a good way) just from changing the model, but neither version reproduced the original.

At this point, I don’t know whether I’m dealing with:

• something still wrong or different in the underlying architecture/context handling;
• a difference in how the instructions and knowledge are being assembled or passed to the model;
• differences between models or between the original environment and this one;
• or some combination of those things.

And then there’s the Remy cost problem.

My original build conversation became enormous - around 800k tokens of context. I eventually cleared/reset it and had Remy familiarize itself with the updated documentation, but even small investigations now seem surprisingly expensive. Things that sound like simple read-only questions can cost dollars.

So now I’m in this weird position where I need technical help figuring out whether the underlying build is sound, but using Remy to investigate the Remy build is itself getting expensive.

I’m okay continuing to invest in the build if we’re actually finding and fixing root causes. What I really don’t want to do is spend another several hundred dollars or more making increasingly specific patches without knowing whether there’s still something upstream causing the problems.

Has anyone here dealt with something similar after a build required substantial backtracking?

And for those of you with larger builds: how do you audit whether the underlying architecture is actually right before continuing to fix things at the behavior/output level?

Also, if anyone has figured out a sane way to work with Remy once a project has accumulated a huge amount of context without every little investigation getting expensive, I would really love to know what worked for you.

Thank you.