June 3, 2025
I’m not finished telling the story of the prototype! I learned a lot even as I drove into that dead end:
There I was, staring at a classic architectural decision: Do I build the basic features first (like every tutorial says), or do I build the learning infrastructure before I have anything to learn from?
My architect brain (OK, it was Claude Opus — I’m not a programmer!) said something unexpected: “Build the learning scaffolding first. Capture signals from day one.”
This felt backward. Cart before horse. Foundation before… foundation? But here’s what convinced me, but I felt like this was one of those “new” things about LLMs that I had to wrap my mind around. This is supposeed to be software that learns. So when should the learning start?
The middle path
The choice wasn’t actually binary. We didn’t need a full machine learning pipeline with neural networks and whatnot. We needed what I started calling “learning scaffolding” — just enough infrastructure to capture learning signals without over-engineering:
- Event Bus: Every significant action broadcasts an event
- Feedback Capture: Simple thumbs up/down with optional comments
- Learning Signals: Track what knowledge Piper thinks it needs
- Confidence Scores: Know when the AI is guessing vs. knowing
Nothing fancy. Just plumbing. But plumbing that would matter later.
The first real test
Here’s where it got interesting. I gave Piper this request: “Create a new ticket for the login bug affecting mobile users.”
Not only did it classify this correctly (EXECUTION/create_ticket with 0.95 confidence), but it also told me what it wished it knew:
- Project management tools knowledge
- Bug tracking systems
- Mobile development context
- Authentication systems understanding
This list? I didn’t program it. Piper reasoned it needed this knowledge. Without me having taught it anything about product management it was still able to generalize from… having read the whole Internet and a bunch of pirated books, including my own?
Why this matters
Most AI systems are built with intelligence first, learning later (if ever). That’s like building a house and then trying to add plumbing through the walls. Possible? Sure. Fun? Not remotely.
By starting with learning infrastructure:
- Every interaction teaches us something
- Patterns emerge from actual usage
- Users train the system without extra work
- We know what Piper doesn’t know
The technical gut checks
Mind you, building this wasn’t smooth sailing. We hit the usual suspects:
- Import circular dependencies (solution: shared_types.py pattern)
- Async/sync event handler mismatches (classic)
- Services that needed to respect existing architecture
But here’s the thing: these weren’t just bugs, they were the architecture telling us something. When you hit circular dependencies, you’ve found a boundary. When event handlers clash, you’ve found a coordination point.
The aha! moment
The breakthrough wasn’t technical this time. It was philosophical.
Traditional approach: Build features → Add analytics → Maybe learn something
Our approach: Build learning → Add features → Learn while building
What actually worked then
As of this session, the Piper prototype could:
- Process requests with real AI reasoning (not templates)
- Identify its own knowledge gaps
- Capture feedback on every interaction
- Store learning signals for future processing
The feedback loop is complete: intent → classification → learning signals → corrections → storage.
Is it learning yet? Not really. That’s like asking if plumbing is washing dishes. But when we’re ready to add actual learning algorithms, every interaction up to that point will be training data.
The meta-lesson
Building with AI assistance (Claude, Cursor Agent, and friends) taught me something: the tools want to add features. They’re really good at adding features. “Would you like me to implement sophisticated neural network learning?” they ask.
No. I want plumbing. Boring, reliable plumbing that will still work when we swap out the AI provider or change the learning algorithm or discover we’ve been thinking about the problem wrong.
Sometimes the best feature is the infrastructure that makes future features possible. Even if it feels like you’re building backwards.