THE THING WE BUILT:

Open Flavor

“Our chatbot will NOT stop talking about 🌭🌭🌭. Every response, a glizzy reference. Mustard this, Chicago dog that.”

“Why does our cookbot keep talking about hot dogs? What’s going on?”

“…oh. There’s a line in the instructions. I forgot that’s still in there.”

Okay, so maybe I added that hot dog line on purpose for this demo. But this sort of thing comes up all the time now.

In AI for Good, Josh Tyrangiel tells the story of Khan Academy building Khanmigo with OpenAI. The team expected the hard part to be software engineering. It wasn’t. The hard part was watching students talk to the tutor, catching where it failed, and rewriting the prompts, over and over. The instructions encoding how it should teach kept growing and changing. Prompt work felt less like coding and more like curriculum design. There was no permanent “done.”

As chat interfaces get more common, we see it ourselves: testing with our team, testing with clients, hitting an unexpected answer, then having to dig in and figure out why. In a lot of cases that’s much harder to track than a deterministic bug in code. But you can’t answer with a shrug.

We’ve been thinking of experiments that could help close that loop. So we built a thing.

Why does it keep mentioning hot dogs?

How It Works

The chat sits on the right. On the left: the stuff that actually flavored the answer.

It isn’t just one prompt anymore. Real chat products stack layers of standing rules, plus context from the page you’re on. Instructions are voice and behavior: what to always do, what never to invent. Context is the turn’s facts: pantry counts, who’s coming to dinner, the open recipe. Wrong behavior? Edit instructions. Wrong number of eggs? Edit context. Mixing those up is how you get nonsense “fixes.”

When a reply misses, you say what was wrong. We try to route that complaint to the right place (a rule to cut, or a fact to correct) and show it as a plain red/green diff. Apply it. Ask again. Or don’t. Either way, you’re fixing from the same screen where you noticed the failure.

Why this Matters Now

Early on, we talked about “the prompt” as if it were one thing. Now every reply is flavored by layers of instructions, behavior rules, and context. When something goes wrong, the failure still shows up as a weird sentence in a chat, but the cause might be a forgotten line, a bad number, or two rules fighting each other.

But most tools still treat all of that flavoring as configuration: hidden away, edited in a different screen, often buried in a codebase. So when the AI says something weird, the people who are most qualified to recognize what’s wrong often aren’t the ones equipped to fix it.

The closer you can move debugging to the conversation itself, the faster you can teach an AI to behave the way you actually intended. What if the chef could teach the cook without ever opening a repo?

That’s the experiment, anyway.

Built with ❤️ by Township

You’re receiving this because you signed up for Township’s “We Built a Thing” newsletter. We'll only send these when we, well, build things.