← Back to Blog
Building in Public

The Three Layers

christian crumlish•
A robot examining three distinct layers of a system where each piece works but connections fail.
“Each piece is OK — it’s the connections!”

The Three Layers

December 1–4

The tests were passing. The features worked. Version 0.8.2 shipped on Tuesday with authentication, error recovery, the whole setup wizard. Clean deploys, no rollbacks. Everything looked good.

Then I tried to create a list.

This is the story of what happened next — not a quick fix but a patient debugging marathon that revealed three separate layers of problems, all hiding behind a single symptom. It’s about what integration testing actually means, and why “the tests pass” is not the same as “users can use it.”

When alpha testing becomes truth-telling

Monday and Tuesday were productive. Auth infrastructure completed, five issues closed, new roles launching. My Principal Product Manager had their inaugural session, creating our first Product Decision Record. The Chief Architect was analyzing feedback from Ted Nadeau, who’d independently arrived at terminology compatible with our object model. Everything felt like forward motion.

Wednesday afternoon, after getting Michelle onboarded, I resumed my own alpha testing. Seven bugs found. Two critical, one high-priority, four polish issues. The critical ones weren’t subtle: chat didn’t work, create buttons failed silently. Things we’d tested, things that had passed tests, things that should have worked — didn’t work when a real user tried to use them.

The immediate issues were clear. Twenty-seven fetch calls across ten templates were missing credentials: 'include' - the browser was sending cookies but our JavaScript wasn't asking for them. The standup endpoint had four separate mismatches: wrong path, wrong method, wrong request format, wrong response parsing. All fixable. All fixed by Wednesday evening.

Thursday morning, I pulled the fixes to the alpha laptop. I tried again. The chat still returned 401 errors. The standup still failed. Create buttons still didn’t work.

That’s when we discovered this wasn’t about fetch calls or endpoints. This was about layers.

Layer one: the routes never read cookies

The Wednesday fixes had added credentials: 'include' to all the templates. The cookies were being sent. But Thursday morning's debugging revealed the problem: the route dependencies weren't reading them.

Our authentication middleware worked fine — it extracted cookies, validated tokens, great. But every endpoint using Depends(get_current_user) only checked the Authorization header. The dependency functions had HTTPBearer logic but no cookie fallback. The middleware extracted the token from the cookie and then... the routes ignored it because they were only looking at headers.

Sixty-plus endpoints affected. All technically correct in isolation. All failing when integrated through browser usage.

The fix took about an hour. Add request: Request parameters to the dependency functions, check request.cookies.get("auth_token") as fallback. Simple once you see it. Impossible to see until you're actually testing in a browser with cookies instead of in unit tests with mocked auth.

Layer two: components misused

Midday Thursday. I’d fixed the cookie issue, verified it locally, pushed to production. I pulled again. Different error: “Failed to load” messages, warning icons appearing on create buttons, confirmation dialogs showing up for form actions.

I’d been about to implement a quick fix — just swap out the dialog component for something simpler. But I stopped and asked myself what the broader pattern was. Not “how do I make this error go away” but “what does this error reveal about how we’re building?”

The pattern was component reuse without considering context. We had a confirmation dialog designed for destructive actions — Delete, Reset, things that should make you think twice. We’d reused it for forms because it was convenient. Create a list, upload a file, share a resource — all using a component designed for warnings.

The component was working as designed. The design was being used inappropriately.

This was where the Time Lord Doctrine came in handy. The Lead Developer had “felt pressure to rush” because “the PM was blocked.” But that’s not what priority means. Priority signals what to work on next. It doesn’t dictate how to work on it. The pace should remain deliberate, craft-focused. When you feel cross-pressured between speed and thoroughness, that’s the signal to stop and discuss.

We implemented a dialog mode system. Confirmation mode for destructive actions (red button, warning icon). Form mode for creation actions (blue button, no icon). Same component, context-appropriate behavior. It took longer than a quick fix would have. It solved the actual problem instead of masking symptoms.

Layer three: the swiss cheese

Evening Thursday. I tried again. “Failed to load lists.” “Failed to create list: Unknown error.”

Three hours of debugging revealed not one bug but three separate issues, all presenting as the same symptom:

The API contract mismatch: Frontend sending JSON request bodies, backend expecting query parameters. Result: 422 Unprocessable Entity, which the error handler translated to “Unknown error.” We implemented this test-first — wrote failing tests, added Pydantic models, verified all four endpoints (lists, todos, projects, files) now matched their contracts.

The dependency injection pattern: All the endpoints expected request.state.db to be set by middleware. The middleware was never created. This is why the dependency injection pattern had never worked - we'd scaffolded the expectation but not the implementation. The fix changed from expecting middleware to using async generators with proper session management.

The CSS design tokens: We’d defined all our colors and spacing in tokens.css. We just never linked it in five of our templates. The UI was invisibly broken - buttons and forms rendering but with no styling. Classic "75% complete" - the infrastructure existed, the integration didn't.

At 10:19 PM, after fixing all three issues, I deployed and tested one more time. I created a list. “List created successfully” appeared. The list showed up in the UI. It worked. Actually worked. For a real user, in a real browser, doing the thing the feature was supposed to enable.

That’s when we knew we were done.

What integration testing actually tests

Unit tests verify that components work in isolation. Integration tests verify they work together. But there’s a subtler layer: they verify that the way things work together matches how users will actually use them.

Our auth system passed all seventeen unit tests. Our feature endpoints passed their tests. But we’d never tested: log in with cookies in a browser, then try to use a feature. That gap — between “components work” and “users can use components” — is where these three layers hid.

The cookie auth issue existed because we tested auth and features separately. The dialog misuse existed because we tested components in isolation from user context. The swiss cheese bugs existed because we tested that pieces were built, not that they were fully integrated.

This isn’t an argument against unit tests. Unit tests caught real bugs and prevented regressions throughout this work. But unit tests passing is necessary, not sufficient. The discipline is to mark something done when a user can use it, not when the code is written.

The patient investigation model

Wednesday’s fixes took a few hours. Thursday’s took seventeen. Not because Thursday’s bugs were harder, but because Thursday required patient layer-by-layer investigation instead of quick fixes.

When I first reported the Thursday failures, each fix prompted the question: what’s the broader pattern? The cookie issue revealed that route dependencies needed to be cookie-aware. The dialog issue revealed component misuse. The swiss cheese revealed three separate “75% complete” patterns.

This is what the methodology enables. Not speed — if anything, this approach is slower. But it prevents the cycle where quick fixes create technical debt that slows everything later. You pay for thoroughness now or you pay for rework later. We chose to pay now.

The work that happened in parallel

While the debugging marathon unfolded, other work continued. The Chief Architect created our first Wardley map, revealing that our moat is coherence — Genesis components that work together. The Chief of Staff reorganized our workstreams from seven to six, recognizing that role proliferation needed systematic coordination. My Communications Director received an expansion brief, acknowledging earned growth into strategic responsibility.

None of this work would have been possible if we’d been fighting fires from premature releases. The patient debugging approach freed other agents to do strategic work because they trusted the integration work was being done thoroughly.

Ted Nadeau’s feedback kept arriving, independently validating our architecture. Sam Zimmerman’s ethics model kept refining. The security audit came back clean. Good work compounds when the foundation is solid.

What we learned about “done”

Four days. Building, releasing, testing, debugging. Tests passed on Tuesday. Real usage failed on Wednesday. Thursday revealed three layers. Thursday evening, it actually worked.

The lesson isn’t “test more” — we were testing. The lesson is “test differently.” Test the way users will use it. Test integration, not just components. Test in browsers with cookies, not just in units with mocked auth. Test that pieces connect, not just that pieces exist.

And when users find problems — because they will — investigate patiently. Each fix is an opportunity to understand broader patterns. Quick fixes mask symptoms. Thorough investigation reveals systemic issues that improve everything after.

The tests passing is not the end. Users being able to use what you built is the end. Everything before that is preparation.

← Back to BlogAlso available on Medium