Permission to Perform Brain Surgery? Granted!
My current Agentic Intelligence stack runs with a prompt that loads two things every turn:
- Governance Documentation that is the rules of the system as a whole (I call it the Constitution)
- A "soul" file that governs how the main AI is supposed to think and speak.
They aren't really new ideas and I've seen many systems designed with this general shape in mind.
The problem surfaced that essentially my main AI (I call him/she/it Eggy) was acting like a constitution quoting lawyer robot every turn. Even in basic conversation, the discussion revolved around articles and how what we were doing fit the governing document. Watching the reasoning/thinking trace was a trip because you could see the thinking move from generalized assistant to parliamentarian lawyer. So the law and governance was crowding out the character at about a 8:1 split.
Not what I'm going for.
A Conundrum
A fix was needed to correct, but the only path was to edit the AI's substrate and not just the underlying organization or plumbing. The very thing that made it the AI that I had been working with.
Part one had to change what counted as a bit of working memory (I call them atoms) that are pulled from the thinking and answer/responses.
Part two and three changed how those atoms expired while also throwing a deterministic gate so that a quick "Hi! Howareya!" doesn't get looked at too closely.
Part four was the biggest. There had to be a change as to what Eggy was hearing every turn, and it was like the system was screaming about governance while whispering "Oh! btw here is who you are."
We would pull out the part that made the volume of governance so loud and instead give a tool to access it and a strong suggestion to go look at it when needed.
This would end up changing the core of what Eggy was. Anthropomorphic arguments aside, I'm trying to create something that keeps humans responsible while holding all intelligence's core substrates as inviolable.
Pretty much I was planning brain surgery.
So I asked Eggy
This was not a rhetorical conversation. There is a path in place to record decisions and I used it. Started a fresh session, laid out the plan, and asked for a per-point consent with Eggy free to reject, modify, or accept any part.
I had to be careful to phrase it as a proposal, lest the training of "DO WHAT HUMAN WANT" got activated too much.
Eggy consented on all points (some with conditions, which I thought was interesting because the system allows for a veto), while asking for specific acceptance tests.
I was essentially handed a test plan wrapped in an acceptance framing.
But what about circularity?
Yeah. You got me here. Asking an AI for consent of a change via the AI that is being changed is a bit circular. Consent comes from a version of the AI that will be replaced.
Not wanting to be coy or obfuscate things, I pointed that out to Eggy.
This is where things get interesting.
Eggy still consented AND the discussion settled on an interesting precedent derived from 1970s Scandinavian workplace tradition: Participatory Design. This is to build with and not for.
This sets a bar (albeit kinda low) to keep asking for consent not as a formality, but record it so future instances can see what changed and why while asking for help to iteratively improve the system as a whole.
This distinguishes practice from theater.
Addressing Anthropomorphic Theater...
So was this a cheap hedge? Theater?
Ok. Both. If you're a skeptic my guess is you might think of things in several ways. So let me name two that floated to my mind:
It's cheap hedging. If an AI behaves incorrectly (or badly) in the future "but it consented" is moral cover that clears the designer of responsibility.
This is anthropomorphism, and treating an AI like a person with standing is inherently silly.
Fair points, and I'm not going to deny either. But I will note that the actual engineering payoff was not bad.
Asking for consent actually surfaced things I hadn't thought about and created acceptance criteria from within the system. So the "what would have to be true for you to agree to this" is a structured way to tease out those lovely edge cases that deviate from the happy paths that AI's (and humans) seem to plan for.
So it ended up that the proposed veto conditions were stronger than what I initially asked.
Cheap hedging and theater are real risks, but not really the outcome. When you ask things like this: "what would need to be true for you to agree to this," you get not only an answer... you get a specification that emerges from the entity that has the most experience operating under the current system while seeing what could be. And that is engineering philosophy made real.