The Drift We Didn’t See
February 1
I was in the middle of a debugging session, trying to figure out why todos weren’t working as expected, when I noticed something in the terminal output I didn’t understand (“timestamp-without-timezone warnings”), so I asked what it was about.
What we found over the course of the day would touch 80+ files, update 47 database columns, and reveal infrastructure drift that had been silently accumulating since the beginning.
The first thread
The timestamp warnings led to creating an issue for “Schema drift — DateTime vs timestamptz mismatches.” I recalled we had refactored something about timestamps a while back where I was seeing GMT time instead of my local Pacific time, and it sounded to me like another case of unfinished business.
The fix seemed simple at first: use UTC-aware datetime comparisons. But fixing one comparison revealed the next problem.
The widening gyre
Once we started looking for timezone drift, we found it everywhere: 239 utcnow() calls needed updating. Each investigation revealed adjacent issues. The scope kept expanding. We ran a systematic audit. The answer: 47 database columns using timestamp without time zone when they should have been timestamptz. Dozens of code paths using datetime.utcnow() when they should have been using timezone-aware functions. The drift had been there since the beginning, invisible because it was everywhere.
The schema migration
The fix required touching the database schema itself. This always scares me a bit, but I remind myself, we’re still in alpha! We have no real users to inconvenience.
We updated the 47 columns. Every timestamp without time zone converted to timestamp with time zone. Every stored datetime now explicitly UTC.
The migration ran cleanly — PostgreSQL handles the conversion gracefully when the data is already effectively UTC. But the migration was only half the fix. The code still needed to use UTC-aware operations.
We created datetime_utils.py—a utility module with functions for timezone-aware datetime handling: utc_now() for the current time in UTC, ensure_utc() to normalize incoming datetimes, is_timezone_aware() to validate. Then we updated every call site. 80+ files. Every datetime.utcnow() examined. Every comparison audited.
What drift looks like
Timezone drift didn’t announce itself. There are no error messages, no stack traces, no logs saying “WARNING: comparing apples to oranges.” The code runs. The comparisons complete. The results are wrong. Sometime in hidden ways and in other cases more blatantly. I’d get shown tomorrow’s calendar because it’s already tomorrow in the UK, for example!
This is the insidious nature of infrastructure drift. The system keeps working. The tests pass — because the tests were written with the same assumptions as the code. The bugs hide in the gap between “runs without errors” and “produces correct results.”
We only caught it because warnings appeared during an unrelated debugging session. If I hadn’t noticed them scrolling by in the terminal, or hadn’t asked about them, we might have run for months with subtle datetime mismatches, gradually accumulating invisible issues.
The cascade pattern
Each investigation revealed adjacent issues. That’s not a bug in the methodology — it’s a feature. Pulling one thread reveals others. The alternative is fixing bugs one at a time, never seeing the systemic pattern, patching symptoms while the root cause persists.
The work also led to discovering 3 pre-existing test failures that were “not related to this issue but that we then fixed afterward. The investigation that led to a net improvement in the codebase and not just a remediation.
The todo debugging led to noticing warnings led to the schema audit led to the 47-column migration. Each step was discovery, not scope creep. We weren’t adding work — we were finding work that was always there, hidden by its own consistency.
One day of investigation. 80+ files modified. 15 new tests for the datetime utilities. One comprehensive schema migration. The drift we didn’t see is now the infrastructure we can trust.
What we couldn’t have known
The timestamps were set up this way at the beginning. Naive datetimes are the Python default. timestamp without time zone is a valid PostgreSQL choice. Each individual decision was reasonable. The drift accumulated silently, one reasonable decision at a time.
This is why infrastructure audits matter. Not because the original decisions were wrong, but because small choices compound. A timezone here, a naive datetime there — each one fine in isolation, together creating systemic drift.
We didn’t know to look for timezone problems until warnings appeared during unrelated work. Now we have explicit UTC handling, schema enforcement, and utility functions that make the right choice the easy choice. The next developer won’t have to discover this the hard way.
The test that passes
One day, 47 columns, 80 files. That’s the math of infrastructure drift: a small symptom reveals a systemic problem. The fix is never as contained as the symptom suggests.
But now the drift is visible. Now it’s fixed. Now the timestamps mean what they say. And one more piece of Piper Morgan is factual, reliable, and stable.