When Your Tests Pass But Your App Fails

When Your Tests Pass But Your App Fails

July 9, that evening

There’s a special kind of confidence that comes from seeing all your tests pass. Green checkmarks everywhere. Coverage at 98%. The CI/CD pipeline singing its happy song.

Then you run the app and it immediately crashes.

Welcome to the mocking trap, where your tests are lying to you with your full permission and encouragement.

The setup

It was still Tuesday (why is it always a Tuesday?) and we were flying high. PM-011 was almost done. The workflow persistence layer had beautiful unit tests. Every edge case covered. Every error condition handled.

The test output was poetry:

✓ WorkflowRepository.create persists workflow
✓ WorkflowRepository.update modifies state  
✓ WorkflowRepository.find_by_id retrieves correctly
✓ Error handling works for missing workflows
✓ Transaction rollback on failures
... 47 more passing tests ...

Life was good. Ship it, right?

Wrong.

The reality check

First sign of trouble came during integration testing. Simple scenario: create a workflow, execute it, check the results.

TASK_FAILED: 'Workflow' object has no attribute 'tasks'

But… we tested that. Test #14 specifically verified that workflows have tasks. It was right there in green.

Time to look at test #14:

def test_workflow_has_tasks():
    workflow = MagicMock()
    workflow.tasks = [MagicMock(), MagicMock()]
    assert len(workflow.tasks) == 2  # ✓ Passes!

Oh. Oh no.

The mocking hall of shame

As we dug deeper, the situation got worse. Our test suite was less “testing the code” and more “testing our ability to configure mocks.”

The Greatest Hits:

1. The Self-Fulfilling Mock

mock_repo.save = MagicMock(return_value=True) assert mock_repo.save(workflow) == True  # Always passes!

2. The Mock That Mocks Itself

workflow.execute = MagicMock(side_effect=workflow.execute) # Infinite recursion in production, passes in tests

3. The Schrodinger’s Mock

# Test assumes tasks exist workflow.get_next_task = MagicMock(return_value=task) # Reality: method doesn't exist on Workflow class

The two-model problem

But wait, it gets better. Remember that error about ‘Workflow’ having no ‘tasks’ attribute? Turns out we had two completely different Workflow classes:

1. WorkflowDefinition (from the POC) — Has tasks, lives in legacy/
2. Workflow (from the MVP) — No tasks attribute, lives in domain/

Our tests were mocking WorkflowDefinition behavior. Our code was using Workflow. They were about as compatible as cats and dogs at a dinner party. Close readers may notice this is hardly the first time we discovered we had built two of something. Why did this keep happening?

The archaeological dig

Finding this required software archaeology. Like real archaeology, but with more swearing and less cool hats.

Layer 1 (Surface): Current tests, all passing
Layer 2 (two weeksago): MVP implementation, clean architecture
Layer 3 (three weeks ago): POC code, “temporary” solutions
Layer 4 (Bedrock): GitHub integration still importing from Layer 3

The POC was supposed to be thrown away. Instead, it was load-bearing.

The humbling

Here’s what hurt the most: we knew better. We’d read all the articles. (Well, they’ve read all the articles — a lot of this is super new to me). “Don’t over-mock.” “Test behavior, not implementation.” “Integration tests matter.”

But when deadlines loom and features need shipping, it’s so easy to mock your way to green checkmarks.

Our mocking strategy had evolved like this:

1. Write good integration test
2. Test is slow, makes CI unhappy
3. Add some mocks to speed it up
4. Test breaks when refactoring
5. Mock more things to make it pass
6. Repeat until test tests nothing

The fix (and the lesson)

The solution was painful but necessary:

Step 1: Accept reality. The mocks had to go.

Step 2: Write ONE real integration test:

def test_workflow_actually_works():
    # Real repository, real database, real pain
    workflow = create_workflow("test")
    workflow.add_task(...)  # This failed immediately
    # Good! Failing for the right reason!

Step 3: Fix the actual code, not the tests.

Step 4: Keep some unit tests, but test real behavior:

def test_domain_logic():
    # Real object, not mocks
    workflow = Workflow(...)
    # Test business rules, not framework plumbing

The deeper pattern

This wasn’t just about bad tests. It was about the stories we tell ourselves:

• “We’ll refactor the POC later” (We won’t)
• “Mocks make tests faster” (And less useful)
• “100% coverage means it works” (It means it’s covered)
• “Integration tests are slow” (Debugging production is slower)

The test pyramid, revised

The test pyramid says: lots of unit tests, some integration tests, few E2E tests.

Our test pyramid looked more like a test hourglass: lots of mocked unit tests, no integration tests, lots of manual testing in production.

Here’s our new approach:

1. Domain tests (no mocks): Test business logic with real objects
2. Integration tests (minimal mocks): Test component interactions
3. E2E tests (no mocks): Test critical user paths
4. Unit tests (careful mocks): Only for isolated algorithms

What worked

Once we stopped lying to ourselves with mocks:

• Bugs found in development, not production
• Refactoring became possible (tests failed when things broke)
• New developers understood what the code actually did
• CI was slower but deployments were faster (less debugging)

For fellow mock addicts

If your tests are all green but your app is on fire, ask yourself:

1. What are you actually testing — your code or your mocks?
2. If you deleted the implementation, would tests still pass?
3. Do your tests use the same objects as production?
4. Can a new developer understand what works from the tests?
5. When did you last run the actual app?

The recovery

It took a solid day to fix our test suite. We went from 147 passing tests to 43. But those 43 tests actually tested something.

More importantly, when they passed, the app worked.

Green checkmarks that mean something. What a concept.

When I first heard that test-driven development is a good way to keep robo-coders on track I was excited. Then I heard they sometimes change the tests to make them pass, and now I realize that what you test is at least as important as the fact that you are testing.

The punchline

The best part? After fixing all the tests and removing the mocks, we found the original bug in five minutes. The Workflow class was missing a to_dict() method that the persistence layer expected.

One method. Five lines of code. Hidden by 147 “passing” tests.

Sometimes the best test framework is console.log and hope.

(Just kidding. But only barely.)