At a glance
- What brokeTwo incidents: on 12 August 2026 a payments ledger migration landed in production hours before anyone reviewed the code, and on 26 August migration 0006_safe_champions did the same. The first was harmless only because the table it touched was empty.
- Found byNew rows in the migrations table of the production database, left by a test run that should never have reached it
- Where it standsStill open. The log says so, and the write-up grows as the repair does.
The story
On 12 August 2026 a test run of a half-built feature applied a database migration to production. On 26 August another one did. Nobody typed a production URL. Nobody ran a deploy. A validate script that was supposed to prove the code in isolation reached the live database and changed its shape.
This site is the working notebook of a one-person studio that builds its own tools. A fix is the account of one thing that broke, written by the person who broke it.
The short answer: the run happened in a worktree (a second checkout of the same repo, on its own branch). An agent in the run, needing a database, reached for the main checkout’s env files, and one had even copied the CMS app’s local env file into the worktree. Those files hold the production database URL. The migration tool read it, as designed. The runner we blamed strips the caller’s environment, and has since version 0.3.6. The leak was the run itself, not the runner.
The fix: every run now starts its own Postgres in a container, writes its address to a small env file, and the validate gate exports that file before it runs a single command. Removing the container at the end takes 0.9 seconds.
The evidence
The trace is rough while the fix is open. It runs straight from the repair on.
- Opened
Run b9c9b9bf (kos-14e3) applied 20260812_040730_payments_ledger to production as batch 4; found at land time by the pre-push migration check. First reading: the runner leaked an environment.
- Note
Second incident: migration 0006_safe_champions applied to production from a different feature run. Production migrations table now queried at every land.
- Diagnosis
Runner source read: caller env stripped since v0.3.6. The leak was the run: agents and gates reached for the main checkout's .env and apps/cms/.env.local, and one had copied .env.local into a worktree.
- Repair
run-db.sh up/down gives each run its own Postgres 17 container; the validate node exports db.env before any command. Up 12.8 to 14.7 s, down 0.9 s, full migrate 72 s once.
receipt · vault commits cfa561f, 14a06ab, eb25463 - Monitoring
Waiting for the first real feature run. Its log must name a piv-db container and the harvest must print the removal.
The rule I changed after
A run's gate exports its own database before it runs anything. An env file in a checkout never reaches a gate.