Puma Slot Exhaustion: Why CPU Sat at 21% During 60,000 502s

This is where to look when a Rails app has spare CPU and memory but keeps throwing 502s and 504s. By the end you can work out how many requests your app can serve at once, and check whether your autoscaling rules watch the resource that actually runs out.

Short version: the bottleneck was Puma's thread slots. Slots are neither CPU nor memory, so no dashboard showed them.

The dashboards looked fine

It was right after a bulk email send. The ALB (AWS load balancer) returned 62,684 502/504s over 50 minutes. Yet nothing on the graphs had run dry.

CPU            normal 1.6%  → peak 21.3%   (2 vCPU)
Memory         66.5%        → 36.7%
DB connections 10.7         → 13.7         (pool 25)
RDS CPU        4.3%         → 9.7%
Response time  0.009s       → 2.7s

Memory actually went down. Autoscaling never moved. I suspected the database first, but RDS stayed under 10%.

Splitting the 5xx in two

The ALB has two 5xx metrics: the ones your app returned (HTTPCode_Target_5XX_Count) and the ones the ALB made up itself (HTTPCode_ELB_5XX_Count).

Target 5XX                       0
ELB 5XX                     62,684
TargetConnectionErrorCount  19,292

The app produced zero errors. Requests never reached it. TargetConnectionErrorCount counts how often the ALB could not even open a connection to the backend. At this point the story narrows to "the app was too busy to open the door."

Twenty slots for 296 requests a second

How many requests a Rails app handles at once comes down to one multiplication.

ECS tasks × Puma workers × threads

That project's puma.rb read WEB_CONCURRENCY and RAILS_MAX_THREADS, but the ECS task definition set neither. The defaults, 2 and 5, went straight in.

2 tasks × 2 workers × 5 threads = 20

Peak traffic was 296 requests per second. Little's Law (requests in flight = requests per second × average response time) gives the concurrency you need: 296 × 2.5 s ≈ 740. That's 740 people lining up for 20 seats.

Why it didn't stop at slow: the health check

Once every slot is busy, new requests wait in a queue. So far that's just slowness.

The catch is that the ALB health check uses the same Puma slots. When the health check times out at the back of the line, the ALB decides the task is dead and stops sending it traffic. Average healthy hosts fell from 1.0 to 0.1. With nowhere to send requests, the ALB answers with 5xx on its own. That's the point where overload turns into an outage.

Why autoscaling can't catch it

The scale-out rules were CPU 80%, memory 80% and a Sidekiq queue of 300. The thing that actually ran out, thread slots, wasn't in any of them. Three policies were attached and wired correctly. They just watched a different resource from the one that dried up. "Configured" and "works" turned out to be different things.

CPU stalling at 21% has a reason too. Ruby has one GVL per process (a lock that lets only one thread run Ruby code at a time). Threads give up their turn only while they wait on I/O. When most of the work is waiting (sending mail, external APIs, slow queries), slots fill up while the CPU idles. "Slots full, CPU at 21%" is the usual shape of an I/O-bound load.

So the right scaling signal is TargetResponseTime. Time spent waiting in the queue is part of the response time the ALB measures. RequestCountPerTarget is a poor stand-in. Normal traffic was 70 requests per 10 minutes at 0.009 s each. During the outage it was 177,514 per 10 minutes at 2.5 s each. With the same request count, the slots you need can differ by hundreds of times depending on how long each one takes.

The trap when adding slots: Puma isn't the only reader of RAILS_MAX_THREADS

My first thought was "just set RAILS_MAX_THREADS=8." Before doing that I searched for everything that reads the variable, and Puma wasn't alone. The database.yml Rails generates sizes the connection pool with the same variable. The repo this blog runs on does too.

# config/database.yml
pool: <%= ENV.fetch("RAILS_MAX_THREADS", 5) %>

That project ran Puma and Sidekiq in one container under supervisord. ECS environment variables apply per container, so Sidekiq got the same value.

Puma threads          5 → 8    intended
DB pool              25 → 8    not intended
Sidekiq concurrency  10 → 10   fixed in sidekiq.yml
→ 10 Sidekiq threads fight over 8 connections → ConnectionTimeoutError

Looking at Puma alone, pool ≥ threads holds and it seems safe. You only see the problem when you look at the other process in the same container. So we split the pool into its own variable, something like DB_POOL_SIZE. One variable doing two jobs was the cause, so tuning the value would just bring it back.

Deploy order matters as well. terraform apply changes the task definition's environment right away, but database.yml lives inside the image. Apply before the pool split ships, and the pool drops to 8 that instant. Infra and code reach production by different roads at different speeds. When two changes are a pair, write the order in the PR.

What to check in your own app first

Start by checking whether WEB_CONCURRENCY and RAILS_MAX_THREADS are actually set in production. If you're running on defaults, you have fewer slots than you think. Puma.stats (or pumactl stats) shows busy_threads, the threads doing work, and backlog, the queue length. On a normal day both sit near zero and everything looks roomy. Which resource runs out first under load is something you only know if you calculate it ahead of time.

The diagnosis checklist and the workers vs threads comparison are in the details below. If you're dealing with a "resources to spare, still 5xx" outage, or want someone to look over a Rails-on-ECS setup with you, feel free to ask negabaro.