Development

When a Coding Agent Writes the Migration: Gates and Dry Runs

Antonio Evans10 min read

A staging database console showing an ALTER TABLE statement with an ACCESS EXCLUSIVE lock badge, a lock-wait timeline chart with a long red bar, and a sign-off panel listing the agent's reasoning.
Key points

Key points

  1. 1An agent-written migration can pass SQL review and still stall the whole table if it waits on an ACCESS EXCLUSIVE lock behind a long-running query.
  2. 2Run every candidate migration against a production-sized copy first and record the wall-clock time each strong lock is held, plus what queued behind it.
  3. 3A lock timeout of a few seconds turns a potential outage into a failed, retryable statement, because it stops the migration from sitting at the head of the lock queue.
  4. 4A named human should approve the measured lock duration and the blocking evidence, not the SQL text alone.

The failure mode: correct SQL, wrong lock

An agent that writes a migration is usually good at producing SQL that says what you meant. It adds the column, creates the index, backfills the field, adds the constraint. The statement parses, the table ends up with the right shape, and in a test database with a few thousand rows it finishes in milliseconds. Everything a language model can see looks fine.

What it cannot see is the state of your production tables. The failure mode that matters is not a wrong migration. It is a correct one that needs a strong lock and cannot get it, so it waits, and while it waits every other statement on that table waits behind it.

Two facts make this happen. First, most ALTER TABLE forms take an ACCESS EXCLUSIVE lock, the most aggressive table lock PostgreSQL has, which conflicts with all other lock modes including the ACCESS SHARE that a plain SELECT takes; the lock level is the strictest one required by any subcommand in the statement (PostgreSQL explicit locking docs). Second, PostgreSQL queues lock requests in order. A migration waiting for its lock blocks later requests that conflict with it, even though those later requests do not conflict with the long query actually holding the lock (the Xata writeup on the lock queue).

The measurable version of this is worth seeing once. In a controlled run on PostgreSQL 18.6, a report query held ACCESS SHARE inside an open transaction, a migration waited on it, and a separate reader queued behind the migration, with pg_blocking_pids() returning the migration's PID rather than the report's. A query on an unrelated table returned immediately (the pg_locks walkthrough). That is the whole outage in four rows: nothing was locked for long, and the table was still unavailable.

A human reviewer reading the SQL cannot see this either, because the cause is a concurrent session on production, not the text. That is why the review has to move from the text to measured behavior.

What the agent should hand you, not just SQL

Change the definition of "done" for an agent-authored migration. The agent is not finished when the file exists. It is finished when it produces four artifacts you can review:

ArtifactWhat it containsWhy you need it
Migration SQLThe DDL and DML, split so each statement touches one tableAvoids multi-table transactions that deadlock against normal traffic
Lock annotationPer statement, the lock mode and whether it blocks reads, writes or bothReviewer sees the risk without re-deriving it
Staging run logTimings on a copy at production scale, lock waits observed, anything queuedTurns claims into numbers
Rollback pathThe exact reversal, or "forward fix only" with justificationDecides whether the change is reversible before it runs

The lock annotation is where the agent earns its keep. Ask it to list, for every statement, the lock the statement requests and the lock times measured in staging. It can also flag the classic traps in its own output. A column added with a constant default is catalogue-only since PostgreSQL 11, but a volatile default such as now() or gen_random_uuid() forces a full table rewrite under ACCESS EXCLUSIVE (RunxBuild on adding columns safely). A plain CREATE INDEX takes a SHARE lock that blocks INSERT, UPDATE and DELETE for the duration, while CREATE INDEX CONCURRENTLY takes SHARE UPDATE EXCLUSIVE and lets writes continue (Bytebase on CREATE INDEX CONCURRENTLY).

If your team runs the agent from a coding harness, a skill that encodes these rules saves you from restating them in every prompt. Quanta Skills is the library of those skills, installable into a coding agent at https://quantaintelligence.ai/skills.

Staging runs: sizing the copy so the numbers mean something

A staging run against a table with a thousand rows tells you the migration works. It does not tell you how long a rewrite takes, because the answer scales with row count and with index count. Size the copy deliberately.

Worked example. Suppose orders holds 40 million rows and carries five indexes. You plan to add status text DEFAULT 'pending' NOT NULL and an index on (status, created_at). The default is constant, so the column addition itself is catalogue-only and fast regardless of table size (RunxBuild). The index is the expensive part. If a full index build reads the table at roughly 200 MB/s in your environment and the heap plus existing indexes come to 60 GB, expect around five minutes of scanning, plus sort and write time, on top of the 40 million row copy. A staging copy that has the same row count and the same indexes, but maybe fewer partitions, gets you a number in the right order of magnitude. A thousand-row copy gets you a lie.

Practically:

  1. Restore a recent production dump into a staging instance with matching instance class and storage. Row counts and index definitions matter more than data freshness.
  2. Run the migration exactly as the agent wrote it, with no manual edits, so you are measuring the artifact under review.
  3. Time each statement separately, not the whole file. \timing in psql or driver-side timing per statement works.
  4. While the migration's strong-lock statement runs, fire a trivial write (INSERT INTO orders ... or an update of one row) from a second session every second and log latency. That series is your user-facing impact, not the migration's own duration.
  5. Record the peak number of sessions waiting on the table during the window.

The last two steps matter because the migration's own runtime and the blocking it causes are different numbers. A CREATE INDEX CONCURRENTLY can run for ten minutes while writes continue, and a non-concurrent build can run for ten minutes while every write to that table queues. Same duration, very different outcome.

For MySQL and MariaDB shops the equivalent numbers come from online schema change tooling. pt-online-schema-change copies the table into a shadow table, records changes with triggers, and swaps at the end, and by default it refuses to run without --execute, applies lock_wait_timeout=60 and innodb_lock_wait_timeout=1, and pauses when replicas fall behind (Percona Toolkit docs). Vitess documents the same shape of problem, direct ALTER TABLE blocking reads and writes, with managed online DDL as the recommended path (Vitess schema change docs). MariaDB's own online schema change copies rows while buffering concurrent changes and acquires a short EXCLUSIVE lock at the end to synchronize (MariaDB docs). The staging question is the same in every case: how long is the window in which writes are blocked or buffered, and how large is the buffer.

Lock-timeout previews: measuring the queue, not just the statement

A lock timeout is the single highest-value line in any migration against a busy table. Set to a few seconds, it makes the migration fail and roll back rather than sit at the head of the queue (Citus on lock timeouts, Xata).

BEGIN;
SET LOCAL lock_timeout = '3s';
ALTER TABLE orders ADD COLUMN note text;
COMMIT;

The preview is a staging exercise that answers three questions with numbers. How often would this migration have failed with a 3 second timeout? What is the oldest transaction on the target table at the times you would deploy? How long does the lock actually get held when it succeeds?

A concrete way to run it. On the staging copy, open a session and hold a transaction with a deliberately slow query (SELECT count(*) FROM orders, pg_sleep(30) or similar), then run the migration with the timeout from a second session. It should fail after 3 seconds, and you should record which statement was blocked and for how long it had been running. Loop this at a few timeout values, say 1, 3 and 10 seconds, and at a couple of workload intensities. The result is a decision table:

lock_timeoutStaging successes out of 20 attemptsLongest successful holdVerdict
1s140.9sSafe to retry in a loop
3s192.4sDefault choice
10s208.7sOnly if 8s of blocked writes is acceptable

The point is not the exact numbers, which depend on your hardware and workload. The point is that "this migration is safe" becomes a statement about a measured success rate and a measured worst-case hold, and a reviewer can disagree with it on evidence.

Two more preview checks belong in the same run. First, confirm what is waiting and who blocks it, using pg_blocking_pids() rather than reading pg_locks by hand, because that function reports soft blocks (sessions queued behind a waiting session) as well as hard ones (the pg_locks walkthrough). Second, check for prepared transactions with no PID, which can hold a conflicting lock and show up as a zero process ID, so a naive blocker query returns nothing useful (same source).

During the staging run, poll this and keep the output with the migration:

SELECT blocked.pid,
       left(blocked.query, 60) AS blocked_query,
       blocker.pid             AS blocking_pid,
       blocker.application_name,
       now() - blocker.xact_start AS blocker_age
FROM pg_stat_activity AS blocked
CROSS JOIN LATERAL unnest(pg_blocking_pids(blocked.pid)) AS b(pid)
JOIN pg_stat_activity AS blocker ON blocker.pid = b.pid
WHERE blocked.wait_event_type = 'Lock';

For index builds specifically, pg_stat_progress_create_index reports the current phase (initializing, building index, waiting for writers, validating) along with blocks and tuples done, which gives you a progress signal on a long build instead of a silent wait (Bytebase on CREATE INDEX CONCURRENTLY). One behavioural detail to remember when reviewing an agent's plan: CONCURRENTLY is an instruction at build time, not a property the index keeps, and the result is an ordinary index once it is valid (Stack Overflow answer on CONCURRENTLY). If a build fails, it can leave an invalid index that needs dropping before a retry, which is a rollback step the agent should spell out.

The approval gate: sign off the numbers, not the SQL

An approval gate is only useful if it approves the right thing. Most migration reviews approve the text. Change that.

Make the sign-off item a measurement table, and require one named person to accept it. The gate passes when all of these are true:

  • The staging run was executed against a copy whose row counts are within 20 percent of production and whose indexes match.
  • Every statement that takes a read-blocking lock on a table above a stated row threshold is either rewritten to a non-blocking form or explicitly accepted with a measured worst-case hold.
  • The skip-locked forms, CREATE INDEX CONCURRENTLY, the NOT VALID plus VALIDATE CONSTRAINT sequence and batch backfills, are present wherever the agent could have used them. The multi-step path for adding a NOT NULL column safely is a nullable add, a code deploy that writes the column, a batched backfill, a NOT VALID check, then VALIDATE CONSTRAINT and a cheap promotion (RunxBuild).
  • The measured lock_timeout success rate from staging is recorded, and the deploy script applies the same timeout.
  • Rollback is written down and tested on the copy, including the invalid-index case.

The named approver should be the person who owns the table's availability, not the person who prompted the agent. That single rule changes behavior: when you know you will be asked for the measured hold time, you run the measurement before opening the request.

Two more gate items that catch agent-specific mistakes. First, split the migration so each transaction touches one table. A migration that alters two tables in one transaction can deadlock against ordinary application traffic that reads them in the opposite order, and the standard fix is to split it or to avoid holding strong locks across more than one table (pganalyze on migration deadlocks). Second, if you use a review tool, point it at the staging logs as well as the SQL. Classic lock-monitoring queries using pg_locks joins can miss or misattribute blockers, and the built-in pg_blocking_pids() plus pg_stat_activity is the reliable pair (PostgreSQL wiki on lock monitoring, Crunchy Data on finding the source of the lock).

If you are adopting this process across several teams, the fastest route is often to hand the reviewer role to someone who does this for a living; Quanta Intelligence keeps AI experts for a line of business, and a migration review expert can run the gate while your engineers keep shipping.

A checklist for the agent and the reviewer

Give the agent this checklist as its definition of done, and give the reviewer the last three lines.

#CheckOwnerEvidence
1Each statement touches one tableAgentMigration file review
2No volatile default on existing tableAgentDefault expression listed
3Indexes use CONCURRENTLY, or accepted with measured holdAgentLock annotation
4NOT NULL added via nullable plus backfill plus VALIDATEAgentStep list
5Backfill in batches, committed between batchesAgentBatch size and timing
6lock_timeout applied at deployReviewerDeploy script
7Staging timings and blocked-write latency recordedReviewerRun log
8Rollback written and testedReviewerReversal log
9Named human approves measured numbersReviewerSign-off record

For the retry loop around a short timeout, the pattern is simple: try, catch the lock-timeout error, back off, try again, and give up after N attempts with a clear error. Failing is cheap when the statement rolls back and leaves traffic untouched; failing silently is not, so make the migration runner exit non-zero and page someone if it exhausts attempts.

Where the agent fits after the gate

Once the gate exists, a coding agent becomes genuinely useful for migrations rather than a source of risk. It can generate the split statements, propose the batch size, write the staging scripts, and fill in the lock annotation. It still should not be the one to decide that eight seconds of blocked writes is acceptable on a revenue table. That is a judgment about your business, and the approval gate exists to put it in front of a person.

Quanta Inference gives you one API for text, image and other model types through an OpenAI-compatible endpoint priced in US dollars, which is convenient if your migration bot also drafts release notes, reads staging logs and classifies errors in one pipeline. The gate does the rest.

FAQ

FAQ

Does a fast migration still need a staging run?

Yes, because the risk is often not the migration's own runtime but how long it waits for a strong lock while everything else queues behind it. A migration that would run in milliseconds can still hold up a table for as long as the slowest query ahead of it, and that number cannot be read off the SQL. Measure it against a production-sized copy and record the wall-clock window in which writes were blocked.

Why is a lock timeout required rather than just strongly recommended?

Because without it the migration waits indefinitely at the head of the queue, and every later statement that conflicts with its requested lock waits behind it, including trivial reads and writes. With a short timeout the migration fails and rolls back, leaving traffic untouched, and you retry when the table is quiet. The alternative is an outage whose length is set by whichever query happened to be running when you deployed.

What counts as a measured number good enough to approve?

A staging run against a copy with matching row counts and indexes, per-statement timings, a worst-case lock hold on statements that block writes, and a success rate across repeated attempts with your chosen timeout. Add the oldest transaction age observed on the target table at deploy-like moments. If a reviewer cannot see how long writes were blocked and how often the migration would have failed, the gate has not done its job.

Do I need different checks for MySQL or MariaDB?

The principle is identical but the tooling differs. Direct ALTER TABLE blocks reads and writes, so teams use online schema change tools that copy into a shadow table, capture changes with triggers and swap atomically, with built-in safety pauses for replica lag and server load. MariaDB's native online schema change does the copy itself and takes a brief exclusive lock at the end to synchronize, so the staging question becomes how large the change buffer gets.

Who should sign off an agent-written migration?

The person who owns availability for that table, not the person who prompted the agent and not the agent itself. That approver should review the measured numbers and the rollback plan, and should be able to reject a correct-looking migration because the measured hold is too long. Naming a person makes the measurement happen before the request, which is most of the benefit.

Sources

Sources

  1. Documentation: 18: 13.3. Explicit Locking - PostgreSQL
  2. Schema changes and the Postgres lock queue - Xata
  3. What is blocking my migration: reading pg_locks
  4. Postgres Add Column: The Safe Way on a Live Table - RunxBuild
  5. How to Use Postgres CREATE INDEX CONCURRENTLY - Bytebase
  6. pt-online-schema-change - Percona Toolkit Documentation
  7. Making Schema Changes - The Vitess Docs
  8. Online Schema Change | Server | MariaDB Documentation
  9. When Postgres blocks: 7 tips for dealing with locks
  10. CREATE INDEX CONCURRENTLY is executed but the ... - Stack Overflow
  11. Avoiding deadlocks in Postgres migrations - YouTube
  12. Lock Monitoring
  13. One PID to Lock Them All: Finding the Source of...
About the author

About the author

Antonio Evans
Antonio Evans

Antonio Evans is the founder of Quanta Intelligence and a fractional CTO who builds HIPAA-ready AI systems for health and life-science companies. Before that he was technical co-founder of FriendsEAT for twenty years, CEO of Socialdraft, and a data engineer at IBM. Antonio lives with his dog Santino in Lisbon, Portugal.

All articles