The October Ghost

January 17

At way too early on a Friday, I woke up to an email from Google Cloud Platform: “Your project has been suspended for hijacked resources.” My first thought: someone hacked us. My second thought: OK what did I do now, and how bad is it? My third thought: coffee first, then crisis.

The alert

GCP’s suspension notice was alarming but vague. “Hijacked resources” could mean anything from cryptocurrency mining to spam operations to something worse. The project was locked. Services were down. And I had no idea what had happened.

I spun up a Security Operations agent at 5:38 AM. First task: figure out what was affected.

The answer came back quickly: Gemini API. Google’s AI service. We used it as a backup for certain operations — not our primary path, but integrated into the system. Someone had been using our API key.

The bleeding

By 5:48 AM, we’d confirmed the scope. The Gemini key was compromised. Usage was spiking in ways we hadn’t caused. Someone else was running requests on our dime.

At 6:04 AM, we deleted the key via AI Studio. The bleeding stopped. Whatever they were doing, they couldn’t do it anymore.

But that raised the obvious question: how did they get the key in the first place? We did an investigation

The five whys

Why was the key compromised? Because it was exposed in a committed file.
Why was it in a committed file? Because it appeared in dev/server-startup.log.
Why was the key in a log file? Because httpx logs full URLs, and our Gemini calls included the key as a query parameter.
Why was that log file committed? Because dev/ wasn't in our .gitignore.
Why wasn’t it caught before now? Because we had no automated secret scanning on the repository.

The root cause wasn’t a hack. It wasn’t sophisticated. It was a logging library doing exactly what logging libraries do — recording URLs — and us not realizing that those URLs can contain secrets.

The timeline

We traced the leak to October 16, 2025. Three months of exposure.

Here’s what likely happened: Someone — probably an automated scanner — found the key in our public repository. They started using it for their own Gemini API calls. Google detected the anomalous usage pattern and suspended the project.

In a way, Google’s suspension was a favor. It forced us to discover a vulnerability we didn’t know existed.

The fix

One-layer fixes don’t prevent recurrence. We implemented five.

The immediate fix was adding dev/ to .gitignore. No more development logs in version control.

For detection, we deployed a URL redaction filter to all HTTP loggers. Any URL containing sensitive patterns gets redacted before logging.

For scanning, we ran git secrets --scan-history across the entire repository history, which found and flagged the October commit.

For prevention, we enabled GitHub’s secret scanning on the repository. Future commits with exposed secrets will be blocked.

For storage, we migrated all API keys from Apple Notes (yes, really) to macOS Keychain with proper access controls.

Each layer addresses a different point of failure. Together, they make the same mistake much harder to repeat.

We also filed an appeal and detailed all the measures we have taken. No word yet….

The lesson

The October Ghost wasn’t a sophisticated attack. It was a combination of ordinary behaviors: a library logging URLs (normal), URLs containing API keys (unfortunately common), development files getting committed (easy mistake), no scanning for secrets (oversight), and credentials stored in notes (convenient but insecure).

None of these individually is unusual. Together, they create a three-month window where anyone scanning public GitHub repositories could find and use your API key.

The fix wasn’t heroic. It was systematic. Five layers, each addressing one link in the chain. Any one of them would have prevented or detected the leak.

The meta-pattern

What struck me afterward was how the same methodology we use for debugging code worked for debugging security. Five Whys traced the root cause. Defense in depth guided the multi-layer fix. Verification confirmed the scope. Documentation captures the learning.

Crisis response isn’t different from regular development. It’s just faster and higher stakes. The same systematic approaches apply.

The cost

We lost a few hours to the incident. I expect the project suspension will be lifted for an innocent mistake now well mitigated. I can’t be the only goofball who has made this mistake!

No user data was affected — Gemini doesn’t handle our user data. The financial cost was some unauthorized API usage within our free quota but of course costing Google something.

But we gained something: a security posture we should have had from day one. Five layers of protection that now exist because a ghost from October forced us to build them.

Sometimes the best improvements come from the worst surprises.