From 2% to 87%: The Great Test Suite Recovery

From 2% to 87%: The Great Test Suite Recovery

July 13

There’s a special kind of despair that comes with seeing your test suite report 144 failures and 19 errors out of 204 total tests. A 2% pass rate isn’t just broken — it’s catastrophically broken. It’s the kind of number that makes you question whether you should just delete everything and start over.

The day I wrote this, we climbed out of that hole, ending the day at 87% pass rate (177/204 tests passing). The journey from software disaster to mostly-working system taught me more about test infrastructure than three years of “testing best practices” blog posts.

The foundation: pre-commit hooks

The recovery actually started three days earlier with something mundane: setting up pre-commit hooks. I had been griping about needing to remember to update documentation every time we changed the code and my colleague at Kind, lead engineer Grace Xu, told me about the concept of pre-commit hooks.

Yes, I have worked in technology all my life and am still learning quotidian programming concepts daily.

I asked Claude to set some up and it configured black, flake8, isort, and other code quality tools to run automatically on every commit. It also reminds us to update documentation based on which files we changed.

This seemed like procrastination at the time — why format code when core functionality is broken? But it turned out to be essential. When you’re fixing 144 test failures, you need every bit of consistency and clarity you can get. Code formatting issues become cognitive overhead you can’t afford.

(At least that’s what I told myself. It is also possible I was being a digital magpie and just grabbing and absorbing every shiny new idea into my process.)

The pre-commit setup also revealed something important: we had 318 files that needed formatting fixes. That’s not a small project — that’s a sign of technical debt accumulation that was making everything harder to understand and debug.

The horror of 144 failures

Sunday morning, July 13. Fresh coffee, good intentions, and a test command that delivered soul-crushing results:

======================== 144 failed, 37 passed, 21 skipped, 19 errors ========================

Two percent. The kind of failure rate that suggests fundamental infrastructure problems, not minor bugs.

The failures fell into categories that told a story:

• AsyncSession leaks — Database sessions not being closed properly
• Missing pytest-asyncio configuration — Async tests failing because pytest couldn’t run them
• Fixture drift — Tests expecting fixtures that no longer existed
• Contract violations — Components expecting interfaces that had changed
• Float precision errors — Tests comparing floats with exact equality

Each category represented weeks of accumulated technical debt coming due all at once.

The systematic triage approach

Rather than randomly fixing tests, we took a systematic approach:

Phase 1: Infrastructure fixes (the foundation)

The async session leak discovery: The most critical issue was database sessions not being closed properly in the query intent handler. This was causing cascading failures as tests shared database state.

The fix was surprisingly simple — add proper session management to main.py:

python

# Before: session leaked across requests

# After: proper async session lifecycle management

pytest-asyncio configuration: Half the test failures were because pytest couldn’t run async tests. Adding pytest-asyncio to requirements and configuring it properly eliminated 20+ failures immediately.

Where have I heard this one before? Ay-sink-ee-oooooh!

Phase 2: Fixture rehabilitation

Missing db_session fixture: Nine tests were failing because they expected an async database session fixture that didn’t exist. Creating the fixture in conftest.py unblocked an entire category of repository tests. I still don’t understand why my bots write tests and either don’t create the fixtures for the tests (and then act surprised when they fail) or create them and then “clean them up,” like some overclocked housekeeper.

Orchestration engine test refactoring: Tests were using mock domain models instead of real ones, causing contract mismatches. Updating them to use actual domain objects fixed another cluster of failures. Beware of test mocks! They are needed when you are testing workflows that are not fully built yet, but if they linger beyond when they are needed, they will trip you up.

Phase 3: Contract alignment

Enum case drift: Intent classification tests were failing because they expected different enum cases than the implementation was using. The kind of drift that happens when you’re moving fast and not running tests regularly. (It also seems to happen a lot because either these bots get sloppy or they are mixing up knowledge from programming languages with different conventions.)

Repository interface changes: Some tests expected connection pools while others expected sessions. The infrastructure changes we’d made for better async support had shifted interfaces without updating tests.

The psychological challenge

The hardest part wasn’t the technical fixes — it was maintaining the belief that the situation was recoverable. When you see 144 failures, the rational response is panic. When you fix 10 and still have 134 remaining, it’s easy to conclude you’re making no progress.

The breakthrough came when we started categorizing failures instead of just counting them. Once we could see that most failures were infrastructure issues rather than logic problems, the path forward became clearer.

The momentum shift

Around hour 4 of the recovery session, something shifted. The fixes started cascading. Resolving the async session leak eliminated 15 failures. Configuring pytest-asyncio properly fixed another 20. Adding the missing fixture unblocked 9 more.

Suddenly we weren’t fighting individual test failures — we were addressing systemic issues that affected multiple test categories. The pass rate went from 2% to 30% to 60% to 87% as each infrastructure fix unlocked multiple working tests.

What was still broken (and why that was OK)

The remaining 13% of test failures fall into categories that represent real work, not just infrastructure debt:

FileRepository pool vs session architecture: Tests expect connection pools but the implementation now uses sessions. This represents an architectural decision that needs to be resolved, not just a bug to fix.

API integration contract drift: Some query service tests are failing because the interfaces have evolved during development. These failures are telling us about API design decisions we need to make.

Logic assertion mismatches: Tests expecting specific float values or exact string matches that need to be updated for current implementation behavior.

These aren’t blocking failures — they’re indicators of technical decisions that need conscious resolution.

The infrastructure lessons

The recovery taught us several important principles about test infrastructure:

Async testing requires specific setup. pytest-asyncio isn’t optional when you’re building async systems — it’s foundational infrastructure that needs to be configured correctly from the start.

Database session management is critical. Session leaks don’t just affect performance — they make tests unpredictable and unreliable. Proper session lifecycle management is essential for test stability.

Formatting consistency reduces cognitive load. When you’re debugging complex failures, code formatting issues become mental overhead you can’t afford. Automated formatting is productivity infrastructure.

Systematic triage beats random fixing. Categorizing failures by root cause reveals the high-leverage fixes that unblock multiple tests at once.

The AI assistance reality

Cursor Assistant was invaluable for systematic analysis and fixture creation. But the recovery process highlighted something important: AI is excellent at executing solutions but human judgment is essential for prioritization.

When facing 144 failures, AI wants to fix them all in parallel. This can lead to a monomaniacal bug hunt based on guesswork that easily ends up down infinite rabbit holes.

Human insight recognizes that fixing async session management will eliminate 20 failures, while fixing individual assertion errors will eliminate one failure each.

The most effective pattern: human diagnoses systemic issues, AI executes systematic fixes.

The unexpected benefit

The test suite recovery revealed something we hadn’t appreciated: our architecture is actually quite solid. The failures were mostly infrastructure and configuration issues, not fundamental design problems.

The domain models are working correctly. The repository pattern is sound. The orchestration engine is reliable. The intent classification is accurate. The components we built with careful design are holding up under scrutiny.

That’s… actually pretty encouraging.

Looking forward

87% pass rate isn’t perfect, but it’s functional. (Spoiler: It will get better.)

The remaining failures represent conscious decisions to make rather than accumulated debt to fix. The test infrastructure is now solid enough to support ongoing development without constant firefighting.

More importantly, we’ve established a pattern for handling technical debt systematically rather than reactively. When the next batch of tests starts failing (and they will), we have a process for recovery that doesn’t involve panic and random fixes.

The great test suite recovery is complete. Now we can get back to building features instead of fighting infrastructure.