Small Scripts Win: Building Knowledge That Actually Knows Things

June 8, 2025

Remember that knowledge base I mentioned? The one that was supposed to make Piper Morgan smart about product management? Well, it could store documents. It could search them. But it didn’t actually understand them.

Time to fix that. What happened next taught me why sometimes the best architectural decision is the smallest one.

The problem with dumb storage

After getting the infrastructure sorted (Docker still running, Python finally cooperating), I turned to the knowledge system. The POC had successfully ingested my PM book — 85 chunks of wisdom sitting in ChromaDB, ready to be searched.

But the search results were… mechanical. Ask about “user stories” and you’d get every mention of the phrase, regardless of context. The system couldn’t tell the difference between a definition of user stories, an example of one, or a passing reference in a chapter about something else entirely.

The big architecture temptation

My first instinct was to go big. Add a graph database! Build an ontology! Create a sophisticated relationship mapping system with nodes and edges and maybe some machine learning sprinkled on top!

One of my bot advisors even had the architecture sketched out:

Neo4j for the graph layer
Custom relationship extraction pipeline
Semantic parsing with dependency trees
A whole new service just for knowledge management

It could have been glorious. And it would have taken months.

The small script solution

Then I remembered one of my mottos:

No optional complexity

Don’t build what you don’t need yet. Things are already complicated enough as it is. Sometimes the best solution is the one you can ship today.

Instead of a massive architectural change, we wrote four small scripts:

Backup the existing data (always backup first)
Enhance documents with LLM-based analysis
Update the search to use the new metadata
Test that it actually works

Total lines of code? About 200. Time to implement? One day.

Comedy break: The hanging script saga

Of course, nothing ever works on the first try. My initial “enhancement script” was a thing of beauty — a single monolithic bash script that would transform the entire knowledge base in one glorious run.

It hung. Just… stopped. No error, no output, just a cursor blinking mockingly at me.

Turns out, trying to create a massive heredoc in bash while processing 85 documents through an LLM API is a great way to make your computer question its life choices.

The solution? Break it up. Small scripts. Each one does one thing. Each one you can test. Each one that fails, fails fast and tells you why. This won’t be the last time that breaking something down into pieces turns out to be the only way to get it to work right.

Hey, look at me! I’m learning stuff about how to make a system that can learn!

The “magic” of LLM analysis

Here’s what made the difference. Instead of complex algorithms, I just asked Claude to analyze each document:

def _analyze_document_relationships(self, content, metadata):
    prompt = f"""Analyze this PM document and extract:
    1. Main concepts discussed
    2. Document type (bug report, user story, etc.)
    3. Hierarchy level (1=foundational, 4=specific)
    4. Related keywords
    
    Document: {content[:1000]}..."""
    
    # Let the LLM do the heavy lifting
    analysis = self.llm.analyze(prompt)
    return parse_llm_response(analysis)

Simple. Effective. And it actually understood context.

The NumPy 2.0 disaster

Just when things were working, NumPy decided to “help” by releasing version 2.0 with breaking changes. Suddenly:

AttributeError: np.float_ was removed in NumPy 2.0 release

You know what’s fun? Debugging compatibility issues between ChromaDB and NumPy while your environment is held together with virtual duct tape and good intentions.

Solution: Pin the version. numpy<2.0 in requirements.txt. Sometimes progress means knowing when not to upgrade. (This was neither the first nor last “numpy” problem, though.)

Environment variables strike again

Then came my favorite error of the day:

API key not found

But… I had set it. I could see it in my .env file. It worked in other scripts.

The culprit? The ingestion module wasn’t loading the .env file. One missing load_dotenv() call. Hours of debugging for one line of code. I could swear we had already done that!

This is the glamorous life of AI development, folks.

For want of a dot-env file the shoe was lost….

What we actually built

After all the comedy of errors, here’s what emerged:

Before: “Show me documents about user stories”
Result: 15 documents, some relevant, some just mentioning the term

After: “Show me documents about user stories”
Result: Prioritized list starting with definitional content, followed by examples, with relationship scores based on actual understanding

The system now knew:

Document types (architecture docs vs. bug reports vs. tutorials)
Concept relationships (user stories relate to agile, requirements, and stakeholders)
Hierarchy levels (foundational concepts vs. specific implementations)
Contextual keywords beyond exact matches

All from asking an LLM to think about each document as it was ingested.

The incremental architecture lesson

This session crystallized something important: Get the plumbing built, then gradually enrich.

I could have spent weeks building the perfect knowledge graph system. Instead, I spent a day making the existing system 10x smarter. The fancy architecture can come later — if it’s even needed.

The pattern that emerged:

Keep the working system working
Enhance in place with minimal changes
Test each enhancement before moving on
Document what broke so you remember why

Looking back from a month later I can preview that this foundation for learning and (can I say it) intelligence is starting to percolate.

I’m reminded again that sometimes the best way to advance is one small script at a time. Monoliths seem to be brittle. I can’t help feel for the amazing designers I met at Yahoo ages ago who worked for years on not one but two complete overhauls of Yahoo Groups, only to fail to ship again and again due to middle-management cold feet.

Those poor designers had nothing for their portfolio after years of toiling in Sunnyvale, and the product could have been improved one small state at a time that entire period without alienating longtime users or whatever the suits were afraid of.

Small scripts, big wins

By the end of the session, I had:

A backup script (20 lines)
An enhancement script (50 lines)
An update script (30 lines)
A test script (25 lines)

Each one focused, testable, debuggable. When something broke (and everything broke at least once), I knew exactly where to look.

Compare that to debugging a giganto 500-line script that hangs with no output. I’ll take small scripts every time.

Looking forward

With the knowledge system actually understanding PM concepts, Piper Morgan could now:

Provide contextually relevant information
Understand relationships between concepts
Prioritize foundational knowledge over specific examples
Actually help with PM decisions, not just regurgitate text

Next up: Teaching it to review GitHub issues with this newfound wisdom. But first, I needed to figure out why my virtual environment had decided to forget where Python lived. Again.

The real lesson

Sometimes the best architectural decision is not to make one. Enhancement beats replacement. Small scripts beat monolithic systems. And always, always check if your .env file is being loaded.

The knowledge base could now actually know things. And it only took 200 lines of code, four small scripts, and one bruised ego from fighting with NumPy.

Progress.