Bidirectional Intelligence: Teaching AI to Critique, Not Just Create

June 8–9, 2025

Most PM tools are one-way streets. Create a ticket. Push to GitHub. Move on. But what about all those existing issues that read like someone transcribed a fever dream?

That’s what PM-008 was about: teaching Piper to analyze and improve existing GitHub issues. Turns out, teaching an AI to give constructive feedback is like teaching your kid to critique your cooking. You want honesty, but you really don’t want any lip.

The architectural choice that mattered

We could have built issue analysis as a separate system. New classes, new patterns, new everything. Instead, we built on what we already had:

GitHubAgent: Already knew how to talk to GitHub
IssueContentGenerator: Already knew what good issues looked like
Knowledge base: 85 chunks of PM wisdom from my book
LLM: Could reason about quality if given the right context

By the way, I see I’m again slipping into saying “we decided” when that usually meant Claude (and sometimes my Cursor Agent) proposed something and I said “sure, that makes sense.” Credit where credit’s due.

The URL parsing adventure

First challenge: GitHub URLs come in many flavors:

https://github.com/owner/repo/issues/123
github.com/owner/repo/issues/123
https://github.com/owner/repo/pull/123 (whoops, that's a PR)
owner/repo#123 (GitHub shorthand)

We chose the “accept everything, parse forgivingly” approach:

pattern = r'github\.com/([^/]+)/([^/]+)/issues/(\d+)'

Because in the real world, people paste URLs however they copied them. Fighting user behavior is like fighting gravity — technically possible, but why?

The knowledge context magic

Here’s where it got interesting. Instead of hardcoding rules about what makes a good issue, we had Piper search the knowledge base for relevant PM practices, then use those as context for analysis.

After testing for a while and finding the results, as expected, about as good as a clever college intern could do with the basics of product management down and access to a library. So I asked one of my UX colleagues if I could let my new little bot read one of her github tickets and offer suggestions.

I have yet to meet a UX strategist or designer who loves filling out issues in Jira or Github or what have you, and she had no objection.

Piper wrote this fairly useful comment, which I posted to the real ticket:
This issue provides a comprehensive description of the task to refine the Veteran journey map for choosing a decision review pathway.

The acceptance criteria are well-defined, and the relevant links and artifacts are included for reference.

One suggestion would be to assign a specific team member to the last acceptance criteria point to ensure accountability for documenting and sharing the final version of the journey map.

Overall, this issue is complete and clear, and the team should be able to proceed with the work as described.

Piper wasn’t applying generic rules. It was referencing specific PM practices from ingested documents. Like having a senior PM review every issue, except the senior PM never gets tired or sarcastic.

The confidence score surprise

We added confidence scoring (0.0–1.0) expecting, in this case, that it would generally be high. Surprise: Piper knows when it doesn’t know enough.

Testing on another issue, confidence came back at 0.50. Why? The original issue was so sparse that even with PM knowledge, Piper couldn’t be sure about user impact or technical approach.

This is honestly huge. An AI that says “I’m not sure” is infinitely more useful than one that confidently hallucinates solutions.

Real-world performance

The full analysis pipeline:

Parse URL (flexible regex)
Fetch issue via GitHub API
Search knowledge base for relevant practices
Generate “ideal” version for comparison
Produce analysis with specific improvements

For a test issue:

✅ Identified missing problem statement
✅ Noted lack of acceptance criteria
✅ Flagged absent user impact details
✅ Suggested specific improvements
✅ Generated rewrite with proper structure

All in about 23 seconds. Not blazing fast, but faster than waiting for human review.

Vindicating the composition pattern

Remember that architectural choice to compose (make us of) existing services instead of creating something new? It paid off:

No new dependencies — Everything already installed
Familiar patterns — Same error handling, same logging
Knowledge reuse — PM-007’s enhanced search just worked
Testing isolation — Could test each layer independently

This is what good architecture feels like. New capabilities emerge from existing components without fighting the system.

The humbling reality

Of course, it wasn’t all smooth sailing. Testing revealed:

Python REPL behaves differently than scripts (environment variables, anyone?)
Virtual environments need activation (every. single. time.)
.env files need to be in the right directory
My ability to type “python” instead of “python3” remains inconsistent and I still find myself “fixing” that all the time.

But these are operational grumbles, not architectural flaws. The design held up.

What this enables

PM-008 (when we refactored the documentation and made a roadmap and backlog, all our pending work got numbered) wasn’t just about reviewing issues.

It’s proof that Piper 0.1 could:

Derive quality standards from organizational knowledge
Apply those standards
Provide specific, actionable feedback
Know when it lacks confidence

By now I was impatient to start seeing Piper as an actual conversational partner. I wanted integration with the chat interface! I should be able to sue natural language like “review this issue: [URL]” to trigger the full analysis pipeline.

I also started feeding Piper Morgan’s own backlog issues into it for review. Look, Piper! You’re already helping to build yourself!

The bigger pattern

We’re building bidirectional intelligence. Not just create, but also review. Not just push, but also pull. Not just execute, but also evaluate.

It’s like the difference between a junior PM who only creates tickets and a senior PM who improves everyone’s tickets. Except this senior PM works 24/7 and never gets frustrated by the fifteenth vague bug report of the day.

At this point I still hadn’t reckoned with the fundamental limitations of the prototype. I was still a day or so of manic coding before hitting that wall. At this point I was full of optimism. Four weeks later I would be wondering how this legacy github code with its alternate workflow had made its way into my MVP codebase, forgetting entirely about this brief feeling of a moment of triumph that at least hinted toward the future potential of Piper Morgan.