Why remove_column Breaks INSERTs During a Rolling Deploy
Running db:migrate at boot only protects the new container. Meanwhile old code takes the traffic, and with partial_inserts = false it still writes the dropped column
If you deploy Rails in containers and run db:migrate from the boot script, this is about why remove_column still causes an outage in that setup. By the end you can tell with one console line whether dropping or renaming a column will actually break your app, and ship it safely in two releases.
Short version: migrating at boot only protects the container that is starting up. During the 1 to 3 minutes it takes to migrate and pass health checks, the old containers are the ones taking traffic, and the old code still remembers the column you just dropped.
Why it looked safe
The setup on a work project looked like this.
# Dockerfile
CMD sh /ready.sh && /usr/bin/supervisord -c /etc/supervisord.conf
# ready.sh
bundle exec rake db:migrate
Puma doesn't open its port until the migration finishes, so new code never sees the old schema. That part is true. The mistake was drawing the picture with only one container in it.
The missing piece is the old task
ECS rolling deploys default to minimumHealthyPercent = 100 and maximumPercent = 200. Healthy tasks must stay at 100% at all times, so the new task comes up first and the old one leaves afterwards.
The moment the new task finishes migrating (1), the schema has already changed. Until Puma boots, the health check passes twice and the old tasks drain and disappear (2), every request goes to old code. That's the 1 to 3 minutes.
Running a single server doesn't make it go away. It's about order, not count. With one server, that one server takes all the traffic.
When I ran it, only INSERT broke
To see what an old process actually does, I reproduced it on Rails 8.0.4 with SQLite. The model loads once and caches its column list. Then a separate connection drops legacy_status and renames note to memo. Here's what the same process got back afterwards.
Order.all OK SELECT "orders".* names no columns
Order.where(note: 'n') fails no such column: orders.note
Order.first.note nil no error, just an empty value
o.update!(name: 'b') OK only the changed name is SET
Order.create!(name: 'c') fails has no column named legacy_status
Only INSERT breaks because of partial_inserts. When it's false, the INSERT lists every column the model knows about, not just the ones you set. The log shows legacy_status in there even though nobody assigned it.
INSERT INTO "orders" ("name", "legacy_status", "note") VALUES (?, ?, ?)
ActiveRecord's own default is true. With load_defaults 7.0 or later it becomes false (the when "7.0" block in railties 8.0.4 configuration.rb). The repo this blog runs on uses load_defaults 7.1, so it's affected too. In the same reproduction, switching to partial_inserts = true made the INSERT list only name, and it went through.
So whether you get an outage comes down to one question: does an INSERT hit that table during those 1 to 3 minutes? If yes, every one fails. If not, nothing happens. If you've dropped columns a few times without trouble, those tables probably just didn't get many writes.
rename_column is worse. Any SELECT that filters or sorts on the column breaks as well. Reads return nil without an error, which is arguably worse than a 500: an empty value goes out to the page looking perfectly normal.
Flipping the order just mirrors it
"What if the app goes out first and the migration runs after?" That's how the repo behind this blog works. It deploys with Kamal, and migrations run as db:prepare in a post-deploy hook once the deploy is done. The old container is already gone by then, so dropping a column is actually safe. The new code briefly sees the old schema instead. If new code reads a column you just added with add_column, that gap is no such column.
While that hook was missing, the server's database fell behind schema.rb, and a page that renders blog images returned 500 with no such column: assets.slug. The comment at the top of the hook file still tells that story.
With zero-downtime deploys you can't remove the moment where old code and new schema (or the reverse) live together. So instead of fixing the order, you make the code running in that moment work against both schemas.
The fix: two releases (expand / contract)
Dropping a column gets split like this.
# Release 1: no migration. Make the code forget the column
class Order < ApplicationRecord
self.ignored_columns += %w[legacy_status]
end
# Release 2: now drop it
remove_column :orders, :legacy_status
Release 1 has no migration, so there's no risky window at all. Release 2 still has one, but the old code alive at that point already doesn't know the column. I added ignored_columns to the reproduction and ran it again: after the column was dropped, the INSERT listed only ("name") and passed.
Renames never happen in one go. Add the new column, write to both and backfill, switch reads to the new column, then drop the old one (the two steps above). Four releases. Other cases like adding NOT NULL or UNIQUE are in the table below.
Rules that only live in a doc leak eventually. strong_migrations raises on things like remove_column and rename_column during development. To do it on purpose you wrap it in safety_assured { ... }, and when that shows up in a diff, the reviewer asks "did you split it into two?"
Where to run migrations is a separate question
At boot, as a separate job before the deploy, or in a hook after it: the window above stays wherever you run them. There are other reasons to move migrations into their own job. Migrating at boot also runs when autoscaling adds a task, gets killed mid-migration if it overruns the health check deadline, and can't stop the deploy when it fails. On the other hand, with one task, short migrations and no autoscaling, migrating at boot is the simplest option and good enough. The comparison of all three and the traps I hit moving it to GitHub Actions are in the details below.
What to check in your own app first
ActiveRecord::Base.partial_inserts # false means dropping a column breaks INSERT
grep -rlE "remove_column|rename_column" db/migrate | wc -l # how many times you've done it
grep -rn "ignored_columns" app/models | wc -l # 0 means you never split one
Then check whether migrations run before or after traffic switches. Before: drops and renames are the risk. After: new code reading a freshly added column is the risk. If you have errors that only show up during deploys and can't pin them down, or want someone to plan the deploy order before a column cleanup, feel free to ask negabaro.
How to split each kind of migration
| Change | What happens to old code | How to split it |
|---|---|---|
| remove_column | INSERT fails (partial_inserts = false) | ignored_columns release → drop release |
| rename_column | INSERT, WHERE and ORDER fail; reads return nil | add → dual write + backfill → switch reads → drop |
| Add NOT NULL | INSERT fails if old code writes NULL | code that stops writing NULL → backfill → constraint |
| UNIQUE index | Fails if old code creates duplicates | app validation first → clean duplicates → index |
| Type change | Widening (int → bigint) is usually fine | Narrowing: move to a new column, like a rename |
| drop_table | Every query on that model fails | release that removes the model → drop release |
Where to run migrations: three options
| At boot (ready.sh) | Separate job before deploy | Hook after deploy (Kamal post-deploy) | |
|---|---|---|---|
| What overlaps | old code × new schema | old code × new schema | new code × old schema |
| Risky change | drops and renames | drops and renames | reading a just-added column |
| Good at | Any container from the image has the right schema. One line | Once per pipeline; a failure stops the deploy | Light setup |
| Breaks when | many tasks, autoscaling, long migrations | CI needs permission to start tasks | If the hook goes missing, the DB quietly falls behind |
Four ways migrating at boot goes wrong
- It runs inside the health check deadline. On that project: 120 s grace + 10 unhealthy checks × 30 s = 420 s. Go over and ECS kills the process mid-migration. With
disable_ddl_transaction!the schema is left half-applied - It runs outside deploys. Every time autoscaling adds a task. Any pending migration runs right when load spikes
- A failure can't stop the deploy. Fail → boot aborts → task replaced → fails again. By the time it stops, part of it has already applied
- It runs once per task, at the same time. On PostgreSQL the advisory lock lets one task apply and the rest raise
ActiveRecord::ConcurrentMigrationError. You end up adding a retry loop, and boot gets slower
What tripped me up moving it to GitHub Actions
- ✓With
run-task: true,aws-actions/amazon-ecs-deploy-task-definition@v2runs a one-off task before updating the service and fails the job on a non-zero exit code. No hand-written polling needed - ⚠
aws ecs wait tasks-stoppedgives up after 6 s × 100 checks = 10 minutes. For long migrations use the action'swait-for-minutesinstead - ⚠The action checks the exit code of every container in the task. Reuse the web task definition and sidecars like datadog get judged too, which can block the deploy even when the migration succeeded. Keep a separate migrate-only task definition with one container
- ⚠Expose a
workflow_dispatchthat runs only the migrate job, so you can run release 2 (the drop) after confirming the old code is completely gone
Key Points
Check ActiveRecord::Base.partial_inserts in a console. If false, dropping a column breaks every INSERT in that window
Ship a column drop in two releases: ignored_columns only, then remove_column
Split a rename into four: add, dual write + backfill, switch reads, drop
Block risky migrations at development time with strong_migrations, and question every safety_assured in review
Check whether migrations run before or after traffic switches. Before: drops and renames are risky. After: code that reads a new column right away is