When the Documentation Lied: Seven Blockers Hiding in 1,630 Lines

October 28

Tuesday mornings are always busy and it’s almost noon by the time turn my thoughts to Piper Morgan. The mission: validate that yesterday’s 1,630 lines of installation documentation actually work.

They “work for me in my development environment where everything is already configured” isn’t good enough. They need to work for our first alpha tester (coming soon!) who is not me. Someone with a clean laptop, no project context, following instructions exactly as written.

Time to test the confident note from last night’s wrap-up: “The house is clean for Beatrice Thursday! 🎉”

By Tuesday afternoon: Seven critical blockers discovered. Every single one would have stopped Beatrice cold.

The testing methodology that reveals truth

I begin fresh testing from Step 1, not continuing from Monday’s partial attempts. Clean slate. I test two states first:

No Python pre-installed
Python 3.9 already present (my default Mac setup)

My plan: Follow the guide exactly. Don’t skip steps. Don’t assume knowledge. Don’t “fix” things intuitively. Do exactly what the documentation says.

When something fails: Document the failure. Note what the guide said versus what actually happened. Identify the gap.

This methodology is crucial. When you know the system, you unconsciously work around documentation gaps. “Oh, obviously you need to do X first.” But new users don’t have that context. They do exactly what you tell them.

The testing reveals: We told them wrong things.

Blocker 1: The Python version maze

12:20 PM. Device 1 testing begins with Python installation.

Guide says: Download latest Python from python.org

Reality: Latest is Python 3.14.0

What happens: GUI dialog requests Xcode Command Line Tools (not mentioned in guide)First issue noted: Xcode CLT requirement undocumented.

Python 3.14.0 installs successfully. But there’s a problem.

First terminal: Shows Python 3.9 (system Python)

New terminal: Shows Python 3.14 (correct)

The guide didn’t mention: You need to open a new terminal after installation for PATH changes to take effect.

But the real problem comes later at 4:18 PM:

ERROR: Could not find a version that satisfies the requirement onnxruntime==1.19.2
No matching distribution found for onnxruntime==1.19.2

Root cause: onnxruntime 1.19.2 predates Python 3.14 support. Package has no wheels for 3.14. Won’t install.

The guide recommended “latest Python.” But latest Python breaks our dependencies.

Blocker 2: The repository that doesn’t exist

3:56 PM. Moving to Step 3: Clone the repository.

Guide says: git clone https://github.com/Codewarrior1988/piper-morgan.git

But that is not my repository!

Actual URL: git@github.com:mediajunkie/piper-morgan-product.git

The guide’s URL is completely wrong. Wrong account (“Codewarrior1988” appears to be hallucinated). Wrong repository name. Wrong protocol (HTTPS vs SSH).

I’ve seen Codewarrior1988 before, no idea why Claude keeps hallucinating about it.

This isn’t a minor error. This is “cannot clone the repository at all.” Blocker. Complete stop. Cannot proceed.

Even when corrected to the right account, the HTTPS approach is deprecated, won’’t work. Which means…

Blocker 3: SSH key setup isn’t documented

The guide assumes users have SSH keys configured. What if they don’t?

New users need:

SSH key generation (ssh-keygen)
Adding public key to GitHub account
Understanding host verification prompts
Knowing to type “yes” when asked about host keys

None of this is documented.

At 4:10 PM, I encounter the host verification prompt:

The authenticity of host 'github.com' can't be established.
Host key verification failed

The prompt doesn’t say “press Enter to continue” or “type yes.” New users won’t know what to do.

I type “yes” and press Enter. It works. But only because I finally remembered to pay attention and do that.

This could be another blocker. “What does host key verification mean? What should I type?”

Created comprehensive Step 2b: SSH key generation, adding to GitHub, host verification prompts with exact text and required responses.

Blocker 4: The folder name mismatch

4:16 PM. Repository cloned successfully (after SSH setup).

Guide says: cd piper-morgan

Reality: Folder is piper-morgan-product (from clone URL)

Step 4 command fails. The guide told users to cd into a folder that doesn't exist.

Simple fix: Update guide to use correct folder name.

But revealing: We wrote 1,630 lines of documentation and didn’t test that the folder name matched the repository URL.

Blocker 5: The onnxruntime Python 3.14 incompatibility

4:18 PM. Running pip install -r requirements.txt with Python 3.14.

ERROR: No matching distribution found for onnxruntime==1.19.2

The package predates Python 3.14 support. No pre-built wheels exist. Won’t install.

4:20 PM. Cursor verifies via Context7:

Latest onnxruntime: 1.23.2
Maximum Python supported: 3.13
Python 3.14: NOT SUPPORTED

The guide recommended latest Python. But our pinned dependency doesn’t support it.

4:25 PM. Test Python 3.13 downgrade. New error:

onnxruntime==1.19.2 requires Python >=3.7,<3.11

Wait. Even 3.13 has version constraints with the pinned version?

4:28 PM. Resolution: Test auto-versioning.

bash

python -m pip install onnxruntime

Success! Installs onnxruntime 1.23.2 with Python 3.13 wheels.

Root cause: Version 1.19.2 predates Python 3.13 support. Version 1.23.2 has cp313 wheels and works perfectly.

Fix: Update requirements.txt from 1.19.2 to 1.23.2.

Blocker 6: scipy also lacks Python 3.13 wheels

5:07 PM. Another dependency error:

ERROR: Unknown compiler(s): gfortran
Building scipy 1.13.1 from source

scipy 1.13.1 has no Python 3.13 wheels. Trying to build from source. Requires Fortran compiler that fresh laptop won’t have.

The pattern emerges: Any pinned dependency older than ~3 months becomes a blocker with new Python versions.

Python 3.13 is too new. Package ecosystem hasn’t caught up.

The pragmatic decision: Target Python 3.12

5:10 PM. Decision time.

Option A: Chase Python 3.13 compatibility. Update every dependency. Test each one. Debug version conflicts. Delay Thursday alpha.

Option B: Target Python 3.12 (mature ecosystem). Update guide to recommend 3.11–3.12 only. Document 3.13 as future work.

We choose Option B.

Rationale:

Python 3.12 has mature package ecosystem
scipy, onnxruntime, Pillow all have 3.12 wheels
Chasing 3.13 adds risk without alpha benefit
Thursday onboarding matters more than latest Python

Updated guide: “We recommend Python 3.11 or 3.12. Python 3.13 is very new and some packages don’t have pre-built wheels yet.”

Created GitHub issue: “Python 3.13 Compatibility Migration” for post-alpha work.

Blocker 7: The configuration file that doesn’t exist

5:04 PM. Earlier in testing, another blocker discovered:

Guide says: cp config/PIPER.example.md PIPER.user.md

Reality: Actual file is PIPER.user.md.example

But bigger problem: The guide says to edit PIPER.user.md with API keys.

Reality: System uses OS Keychain for secure storage. NOT plaintext config files.

The services/infrastructure/keychain_service.py implements full keychain abstraction using Python keyring library with "piper-morgan" service name.

Impact: Steps 9–11 in guide are completely wrong. Following them would put keys in config file that system never reads.

This blocker remains partially unresolved Tuesday evening. Needs investigation: How do users add keys to keychain through the setup process?

The guide told users to do something that wouldn’t work. System wouldn’t read keys from where guide said to put them.

What Tuesday’s testing revealed

Let’s count the blockers:

Python version recommendation: Wrong (suggested 3.14, need 3.11–3.12)
Repository URL: Completely wrong (hallucinated account)
SSH key setup: Missing entirely (multi-step undocumented process)
Folder name: Wrong (piper-morgan vs piper-morgan-product)
onnxruntime version: Incompatible (1.19.2 predates 3.13 support)
scipy wheels: Missing (1.13.1 no 3.13 support)
API key config: Outdated (references files/methods not actually used)

Blockers that would stop Beatrice completely: All seven.

Not “minor issues.” Not “polish needed.” But “cannot proceed at all.”

The 1,630 lines of documentation from Monday? Beautiful, comprehensive, detailed, systematic. And fundamentally wrong in seven critical ways.

The live testing model that works

Here’s what Tuesday proved about documentation testing:

What doesn’t work:

Writing comprehensive docs based on your knowledge
Testing in your development environment
Assuming common knowledge

What does work:

Actual human following instructions on clean machine
Immediate feedback loop (report → fix → re-test)
Zero assumptions about prerequisites
Capturing exact error messages and prompts

The testing methodology: Christian reports issue → Cursor fixes → Update documentation → Test again → Repeat.

Tight feedback loop. Real environment. Actual failures.

This is why the blockers got fixed Tuesday instead of discovered Thursday. Because we tested against reality.

Live testing model with immediate fixes… I’m honestly not sure if it’s sustainable at scale but it feels right for these types of issues and alpha prep.

Tuesday’s pattern:

11:24 AM: Start testing
12:20 PM: First blocker (Python)
3:56 PM: Second blocker (repo URL)
4:10 PM: Third blocker (SSH)
4:16 PM: Fourth blocker (folder name)
4:18 PM: Fifth blocker (onnxruntime)
4:30 PM: Fix onnxruntime (commit 3770fa41 pushed)
5:04 PM: Sixth blocker (config file)
5:07 PM: Seventh blocker (scipy)
5:10 PM: Decision on Python 3.12

Approximately one blocker discovered per hour. Each one documented. Most fixed immediately. Some deferred with issues created.

The fixed and the deferred

By Tuesday end of day:

Fixed (5 blockers):

✅ Repository URL corrected to actual GitHub location
✅ SSH key setup comprehensive Step 2b added
✅ Folder name updated to piper-morgan-product
✅ onnxruntime updated 1.19.2 → 1.23.2 (committed and pushed)
✅ Python version recommendation changed to 3.11–3.12 only

Deferred (2 blockers):

⚠️ API key keychain flow (needs investigation and rewrite)
⚠️ scipy 3.13 support (resolved by Python 3.12 strategy)

Testing time: 5 hours 46 minutes of continuous discovery

Five out of seven blockers resolved same day. Two deferred with issues created. Documentation transformed from “fundamentally broken” to “mostly works with known gaps.”

Not perfect. But functional. Ready for Wednesday’s actual wizard testing.

The confidence calibration

Monday ended: “The house is clean for Beatrice Thursday! 🎉”

Tuesday ends: “The house was a mess. We’ve cleaned up five of seven major problems. Two more to go. Maybe the house will be clean for Beatrice Thursday.”

Confidence recalibrated. Hope tempered by reality. Documentation tested and improved.

The 1,630 lines still matter. But they needed correction. Comprehensive doesn’t mean correct.

What changed Tuesday:

Repository URL: Now correct
SSH setup: Now documented
Python version: Now realistic
Dependencies: Now compatible
Folder name: Now accurate

What remains Wednesday:

API key keychain flow (undocumented)
Actual wizard testing (untested with fixes)

The methodology lesson

Here’s what Tuesday teaches about documentation:

Writing documentation isn’t enough. You need to test it. Not “does this make sense to me?” but “does this work for someone who isn’t me?”

Testing in your environment isn’t enough. You need clean machine testing. Fresh environment. No preconfigured tools. No accumulated knowledge.

Comprehensive documentation isn’t enough. You need correct documentation. Better to have 10 lines that work than 1,630 lines that fail.

The gap between “thoroughly documented” and “actually works” is revealed only through real testing.

The lesson that comprehensive ≠ correct seems obvious, but if I’m so smart why do I keep forgetting it?

Tuesday’s achievement: Not writing more documentation. But proving the documentation we had was wrong and fixing it.

The live testing model works: Real user. Real machine. Real problems. Real fixes. Tight feedback loop.

Better to discover blockers Tuesday through systematic testing than Thursday through alpha user frustration.

Wednesday’s challenge

Tuesday ends with five blockers fixed, two deferred, and documentation significantly improved.

Wednesday will test differently: Not “can you follow the docs?” but “does the wizard work?”

The setup wizard that’s supposed to guide users through installation. The automated system that should handle Docker services, database setup, user creation, API key configuration. The one we forgot about when we wrote a guide assuming most of those steps would be manual? Yeah, that one.

Tuesday proved the documentation fails in systematic ways. Wednesday will prove whether the wizard itself works.

[SPOILRT: The wizard will also fail in systematic ways. Three more critical bugs waiting to be discovered through actual setup testing.]

But Tuesday’s lesson remains: Better to find failures through testing than through alpha launch. Better to fix problems Tuesday than explain them Thursday.

The house isn’t clean yet. But we know what needs cleaning and tomorrow is anotherday.