April 8–10
Wednesday evening. Third time at the same gate. We've been throwing ourselves at the "conversational floor" repeatedly and somehow missing. Three lines of code had changed since the last attempt — model identifiers updated, error classifier improved. The CXO had the same scoring rubric, the same nine queries, the same fresh account.
The first query went to the floor. And the floor responded.
After weeks of getting canned responses and generic fallbacks, an actual response, generated by the LLM, informed by context, relevant to what the user had asked. For the first time in user testing, Piper was having a conversation.
I've had a few moments in the past where Piper "spoke" for the first time but this time, with so much of the MVP architecture streamlined and the model so much clearer, it felt more real than ever before.
Five pass, one marginal, two fail
The final tally for round three: five of nine passed the Colleague Test. One scored marginal. Two failed — the affirmation handling bug we'd known about since March, and the GitHub pre-flight check that kept not firing despite multiple "fixes."
But the numbers don't capture what changed. The queries that passed weren't scraping by. "Help me plan a stakeholder presentation" — the query that had first revealed the floor inversion back in March, the question that started this entire investigation — scored eight out of nine. The CXO noted successful multi-turn follow-up. Piper engaged with the planning task, asked relevant clarifying questions, maintained context across turns.
This was the same system. The same codebase. The same architecture. Three characters had changed in a model identifier string, and the experience went from a wall of canned text to an actual working conversation.
The trajectory tells the story
Round one: zero of seven. Round two: zero of nine. Round three: five of nine. Round four, two days later: seven of nine.
Each round narrowed the failure. Round one found five problems — provider config, canned templates, pre-flight checks, todo parsing, input rigidity. Round two proved the provider config fix wasn't enough and forced the Five Whys investigation. Round three confirmed the real root cause and revealed two remaining issues with clear fix paths. Round four verified those fixes.
The Lead Dev found one of those remaining issues — conversation continuity — by tracing why "OK" lost all context. The in-memory conversation model had a field for the user's message but no field for Piper's response. The floor gathered conversation history and only ever saw one side of the dialogue. Piper's own replies were never stored. The code looked correct because the field simply didn't exist to be wrong.
There are days I curse whoever came up with mocks.
What the gate is there for
Gate 2 still hasn't passed. Todo completion has failed in all four rounds. The Lead Dev fixes an apparent cause before each test — regex too rigid, service layer mocked, parser mismatch — and each time the user still can't complete a todo. Pattern-045 resists being fixed because the real failure point keeps hiding behind the apparent one.
But the gate did what gates are supposed to do. It didn't prove the system worked. It proved specifically, measurably, repeatably how the system didn't work. Each round of failure produced actionable findings. Each finding led to a fix that addressed a real problem — even when it wasn't the root cause. The codebase is cleaner, the error handling is better, the architecture is more resilient.
The floor inversion was discovered in March because I sat down and typed a question instead of reading test results. The gate formalized that instinct into a process: fresh account, scoring rubric, specific queries, documented findings, clear pass/fail criteria. No room for "seems to work" or "the tests pass." Either the Colleague Test scores seven or it doesn't.
The floor is alive now. Piper speaks. Not perfectly — there's still a warmth calibration issue, the affirmation bug, the todo completion mystery. But the fundamental architectural bet — that an LLM conversational floor with assembled context would produce colleague-quality responses — is validated.
The MVP sprints are well underway. We've got one more big hill to climb and then we are racing to a release.
Meanwhile, the gate defers to reality and reality pushes back, and you learn more from the pushing than you would have from a clean pass.
Next on Building Piper Morgan: The Drift You Don't Notice, an insight piece from February.
What did your last test failure teach you that a passing test never could?