November 30, 2025
I'd been designing and starting to build an elaborate ethical architecture. Without access to a human board to help adjudicate ethically tricky situations, Claude had proposed a synthetic "board" of specialized agents - an ethicist, a devil's advocate, a boundary guardian - that would collectively establish ethical boundaries through multi-agent consensus. Checks and balances. Deliberation. Governance.
Around that time I bumped into Sam Zimmerman, a brilliant fellow I had the privilege of working with for a while on a Covid-response effort back in that whole [gestures vaguely...] time. Sam's now managing researchers at Anthropic. I know, right? Anyhow, I imposed on him by asking him to read a brief on my approach and his feedback was three sentences that collapsed the whole structure into a clearer paradigm: build ethics from sustained relationship, not committee consensus.
The multi-agent consensus trap
The idea of honing ethical judgment via a multi-agent committee seemed sound on the surface. If one agent might make ethical mistakes, multiple agents checking each other could catch those mistakes. Take "constitutional AI" principles and implement them as a parliament of perspectives. Democratic governance for machine behavior!
Here's the problem, though. Improved ethical judgment is not a matter of avoiding mistakes. It develops through relationship. A committee of strangers can identify violations. Only in relationship can entities develop a shared understanding of what matters and why.
Committees deliberate. Relationships understand. The deliberation model assumes ethics is primarily about decisions - this action or that action, permitted or forbidden. But the deeper challenge is developing context for decisions that haven't been specified yet. That context comes from relationship, not rules.
What relationship-first means practically
For an AI system, relationship-first ethics means:
Shared history matters more than shared rules. The context accumulated through ongoing interaction provides better guidance than explicit policies. Not because rules are wrong, but because rules can't anticipate every situation. Shared history can inform judgment about novel situations.
The relationship contains ethical information. What we've discussed, what's worked and what hasn't, what the person values and needs - this is ethical data. A fresh agent starting from rules has less ethical context than a continuing relationship that remembers.
Trust develops, not just compliance. Multi-agent consensus produces compliance through checking. Relationship produces trust through understanding. Compliance prevents obvious violations. Trust enables appropriate judgment in ambiguous cases.
Why I was fixing to build the wrong thing
Looking back, I adopted the multi-agent consensus proposal because it felt rigorous. Multiple perspectives checking each other. Formal deliberation. Documented decisions. All the things that feel responsible when you're worried about AI ethics.
But "rigor theater" isn't the same as ethical development. I was designing accountability structures where I should have been focusing on relationship, dwelling on governance when I needed to capture accumulated understanding.
Sam's feedback echoed his advice when I ran my ideas about spatial intelligence by him. Keep things simple and trust relationships over elaborate structure. I'm starting to recognize the problems that are solved by building up connections and not by larding on more structure.
Hard limits are non-negotiable
Relationship-first doesn't mean relationship-only. The inviolate boundaries exist precisely because some things shouldn't depend on relationship quality. Hard limits protect against relationship failure.
But hard limits are the floor, not the building. You need the floor - you can't develop ethical relationship without it. But the building is what's lived in. The relationship is where ethical behavior actually happens, decision by decision, interaction by interaction.
My multi-agent board was all floor. Boundaries checking boundaries. No room for the relationship to develop. No mechanism for ethical style to emerge.
My own hard limits include that Piper must never harm people, that Piper must avoid harming organizations, unless doing so would harm people, that Piper must avoid damaging itself, again unless doing so would violate the prior rules, and that Piper is not here to form para-social or pseudo-intimate synthetic person relationships with its users. Icky behaviors ranging from flirting to sexual harassment to other workplace-inappropriate interaction must simply "not computer" to Piper.
Reframing the architecture
I reoriented Piper Morgan's ethical architecture in light of Sam's advice, in a three-layer model:
Layer 1: Simple hard limits, not deliberated. Specified once, enforced always. No multi-agent consensus needed - these aren't judgment calls.
Layer 2: Context-aware adaptation. The system adjusts based on situation, but within hard limits. This requires judgment, but judgment informed by relationship rather than committee.
Layer 3: Ethical style that emerges from sustained interaction. Not designed - developed. The relationship itself becomes the ethical context.
The first two layers can be specified in advance. The third cannot. It has to grow.
This maps to how human ethical development actually works. We have hard limits (don't harm children, don't deceive for exploitation). We have contextual judgment (how direct to be, when to push back, what tone serves the moment). And we have personal ethical style that emerged from our relationships and experiences.
No committee gives us our own ethical style. It comes from sustained relationship with people who matter to us.
This is simpler than my original plan and I hope it will prove more effective as well. I was adding complexity where relationship would serve better.
The crux of the biscuit
Sam understood something I'd missed: multi-agent consensus is still just mechanism, and throwing mechanism at the problem was never going to be enough.
You can't "committee" your way to ethical alignment. Appropriate judgment develops through interaction, through history, through the accumulated context of sustained relationship.
The human advisors helping me to steer this project - Ted, Sam, others - aren't advising me through committee meetings. They're communicating with me in the context of relationship. They know my work, my tendencies, my blind spots. Their advice lands because it comes from shared context, not just external expertise.
Build ethics from sustained relationship. It's simpler than what I was building. And it's more likely to work.
Next on Building Piper Morgan: "From Briefing to Vision" — from reading sixty ADRs on day one to revising the project vision by day eight.
When has simple advice collapsed elaborate plans? How do you distinguish governance-appropriate from relationship-appropriate problems? What role does sustained relationship play in your ethical development?