The Shadow Package Problem

November 19

One file. Four lines of code. Six hundred seventeen tests hidden for months. Zero warning signs until you knew where to look.

A single __init__.py in the wrong directory created a shadow package that blocked pytest from importing the real code, making 617 tests invisible. No collection errors. No failures. No indication that anything was wrong. Just tests silently not running while we assumed everything was fine.

This is the shadow package problem: when infrastructure issues mask themselves as success.

How tests hide themselves

The directory structure looked innocent enough:

tests/
  services/
    __init__.py          # The problem
    test_file_service.py
    test_knowledge_service.py
    ...51 more test files

That __init__.py turned tests/services/ into a Python package. When pytest tried to run tests in those files, it imported from tests.services - the local package we’d accidentally created. But the tests needed to import from services - the actual application code.

Python’s import system prefers local packages over installed packages. So from services import FileService imported from tests/services/__init__.py (empty) instead of from the real services/ directory. The imports failed silently. The tests never ran. Pytest reported zero tests collected.

Zero tests collected looks like “no tests in that directory” not “all tests are blocked from running.” We’d been building features, writing tests, running test suites, seeing some tests pass — never realizing 617 tests weren’t running at all.

The false confidence

This is what makes shadow packages insidious. They don’t fail loudly. They fail silently while looking like success.

If pytest had thrown errors, we’d have noticed immediately. But pytest collecting zero tests from a directory just looks like that directory has no tests. The test suite still ran. Other tests still passed. Everything seemed fine.

We had 68.4% pass rate confidence that was completely false. The tests that were running passed at 68.4%. But 617 tests weren’t running at all. The actual pass rate — if all tests could run — might have been 40%. Or 30%. Or we might have discovered critical bugs the hidden tests would have caught.

False confidence is worse than no confidence. When you know you don’t have test coverage, you’re cautious. When you think you have test coverage but don’t, you’re overconfident. We were making architectural decisions and shipping features based on test results that represented maybe 30% of our actual test suite.

When measuring reveals the problem

The shadow package only surfaced when we tried to measure something specific: architecture test compliance. That test needed to import from both services/ and tests/ directories. The import conflict became visible because the architecture test couldn’t run with the shadow package in place.

Fixing that one test required restructuring the test directory: tests/services/ → tests/unit/services/. Suddenly pytest could import from the real services/ package. And suddenly 617 tests became collectible.

But “collectible” isn’t “passing.” Those 617 tests had collection errors — missing async keywords, wrong imports, syntax errors — that we’d never seen because the tests weren’t running. Fixing collection errors revealed test failures. Fixing test failures revealed actual bugs in production code.

Each layer of measurement revealed the next layer of problems:

Architecture test → revealed shadow package
Shadow package fix → revealed 617 hidden tests
Collection fix → revealed 195 test failures
Failure analysis → revealed systemic issues (missing fixtures, API mismatches, incomplete features)

We couldn’t fix problems we didn’t know existed. The shadow package hid the problems so effectively we thought we had good test coverage.

The broader pattern

Shadow packages are a specific technical issue. But the pattern is universal: infrastructure problems that mask themselves as working systems.

Build systems that skip steps silently — Deployment succeeds, but critical compilation step was skipped, so you’re running old code.

Test suites that don’t run all tests — Pass rate looks great, but you’re not testing half the codebase.

Monitoring that misses critical data — Dashboards green, system healthy, but the metric you actually need isn’t collected.

CI/CD that passes with warnings — Build succeeds with deprecation warnings, tech debt accumulates invisibly.

Logs that don’t capture errors — No errors in logs, but errors are going to /dev/null not to your logging system.

The common thread: systems that fail silently while reporting success. You can’t fix what you don’t measure. And you can’t measure what your infrastructure is hiding.

How to find shadow packages

The technical fix for Python shadow packages is straightforward: move tests from tests/services/ to tests/unit/services/ or tests/integration/services/ to avoid package naming conflicts. Delete any __init__.py files in test directories unless you’re intentionally making them packages.

But finding shadow packages requires knowing they might exist. Warning signs:

Test count seems low — If you have lots of code but few tests, maybe tests aren’t running rather than not existing.

Test count doesn’t grow — If you’re writing tests but test collection stays constant, maybe new tests aren’t being collected.

Import errors in specific directories — If some test directories work but others don’t, directory structure might be blocking imports.

“It works in my environment” — If tests pass on one machine but not others, environment-specific configurations might mask infrastructure issues.

Zero tests collected — Pytest reporting zero tests doesn’t mean no tests exist. It might mean tests can’t be collected.

The deeper insight: assume measurement is incomplete until you verify it’s complete. Don’t trust pass rates without auditing what’s actually being measured.

What fixing shadow packages reveals

We fixed the shadow package and discovered 617 tests. We fixed collection errors and discovered 195 failures. We fixed test failures and discovered missing fixtures, API mismatches, incomplete features.

Each fix revealed the next layer of actual system state. This is good. Revealing problems is better than hiding them. A 68.4% pass rate that’s real is more valuable than an 85% pass rate based on incomplete test collection.

The shadow package created false confidence. Fixing it created accurate data. Accurate data revealed real problems. Real problems got systematic fixes. Systematic fixes improved actual quality rather than apparent quality.

This is the value of infrastructure archaeology. You dig through layers — shadow packages, collection errors, test failures, underlying bugs — until you reach actual system state. Then you can make informed decisions based on reality rather than assumptions.

When to suspect you have shadow packages

Not literally shadow packages (that’s Python-specific). But the pattern of infrastructure masking problems:

When metrics seem too good — Pass rates high, error rates low, performance great — but system still has issues users report.

When measurement is inconsistent — Different tools report different numbers, trends don’t match reality, data doesn’t align with experience.

When new work doesn’t change metrics — Adding tests doesn’t increase test count, deploying fixes doesn’t reduce error rates, improving performance doesn’t change dashboards.

When “it works on my machine” is common — Local environments differ from CI/CD, tests pass locally but fail in pipeline, deployments work in staging but fail in production.

When historical data seems suspicious — Sudden jumps or drops in metrics without corresponding code changes, trends that don’t match development activity, gaps in data that don’t have obvious explanations.

These are symptoms of infrastructure hiding information. Not necessarily shadow packages, but similar patterns where your measurement systems aren’t measuring what you think they’re measuring.

The discipline is: verify measurement completeness before trusting measured results. Audit what’s actually being measured. Test in clean environments to catch environment-specific issues. Investigate when metrics seem suspiciously good.

Because sometimes your tests aren’t passing. They’re just not running. And those are very different problems with very different solutions.