About

You Do Not Turn On ElastiCache Encryption, You Rebuild — Moving Sessions Without Logging Anyone Out, the One-Way DUMP, and the New Endpoint Name

At-rest encryption can only be set at creation. We stood up a -v2 next to the old cluster and moved sessions in two passes. Replaying it on two local Redis servers showed a hole in the copy rule we used

I had to add encryption to an ElastiCache Redis. I expected a checkbox in the console. It turned into building a new cluster and moving user sessions over. After reading this you'll know why a rebuild is required, the order that moves sessions without logging anyone out, and how the endpoint name and TLS check change afterwards.

ElastiCache is AWS's managed Redis (Valkey). There are two kinds of encryption: at-rest, which protects data written to disk and backups, and in-transit, which wraps the traffic between your app and Redis in TLS.

There is no checkbox

The AWS docs say it plainly. At-rest encryption can be enabled on a replication group only when you create it, and you can't toggle it by modifying the group. For an existing cluster the official path is: back up, restore into a new cluster, point the app at the new endpoint, delete the old one.

In-transit can be changed on an existing cluster if it runs Redis OSS 7 or later (Valkey 7.2 or later). Ours was 6.x. We had to rebuild for at-rest anyway, so we upgraded the version at the same time.

If you recreate it under the same name, terraform deletes the old one first and then creates the new one. The names collide, so both can't exist at once, and you get a stretch with no Redis until the new cluster is up. So we built it next to the old one under a different name, -v2, switched the app's connection settings, and moved over with a rolling deploy.

1. Phase 1 before the deploy sessions active in the last 7 days only RESTORE ... REPLACE 2. Rolling deploy old tasks write old new tasks write v2 the gap: both sides change at once 3. Phase 2 copy a key only if it is missing on v2 and was not copied in phase 1 no REPLACE 4. Jobs wait until the last old task is gone then pop-push the leftover queue Why "last 7 days" works Rack expire_after rewrites the session on every request, so the remaining TTL tells you when the user was last seen DUMP is one-way 6.2.6 (RDB 9) to 7.2 (RDB 11): OK 7.2 to 6.2.6: payload version error rollback cannot carry v2 data back

An empty cluster logs everyone out

If sessions live in Redis, switching to an empty v2 logs everyone out. So we pulled values out with DUMP and put them in with RESTORE. The remaining lifetime comes from PTTL (remaining TTL in milliseconds) and is attached as is.

We didn't need to move everything. When a Rack session has expire_after, it is saved again on every request. In the rack-session 2.1.1 source, force_options? looks at expire_after, so forced_session_update? is true and the session is committed each time. The remaining TTL is effectively the time of the last visit.

When we counted, about one in seven of the tens of thousands of sessions had moved in the last 7 days. The rest belonged to people who had closed the browser and left. We only moved the last 7 days.

Two passes

New sessions keep appearing during the move. In a rolling deploy, old tasks write to the old cluster and new tasks write to v2. So we copied twice.

Phase 1  just before the deploy   copy all sessions from the last 7 days   REPLACE is fine
         ↓
         rolling deploy                                                   ← both sides change
         ↓
Phase 2  right after the deploy   fill in only what v2 is missing

In production Phase 1 took about 3 seconds, and Phase 2 filled in a few dozen keys. Copying is fast enough that there's no reason to rush.

Replaying the Phase 2 rule locally

Our Phase 2 rule was "skip if the key EXISTS on v2". To check it, I ran Redis 6.2.6 (old side) and 7.2.17 (v2) in Docker and simulated it with redis-py: 20,000 sessions, expire_after of 14 days, last visits spread evenly over 0 to 14 days. Phase 1 copied 10,099 keys in 0.26 seconds (same Mac, so no network).

Then I staged what happens during the deploy. A changed their cart on a new task, C changed theirs on an old task. E logged out on a new task, and D logged in fresh on an old task. Here is Phase 2 run three ways from that state.

                  A (changed on v2)   C (changed on old)   E (logged out on v2)   D (new login on old)
REPLACE           overwritten         kept                 comes back             copied
EXISTS only       kept                lost                 comes back             copied
EXISTS + list     kept                lost                 stays deleted          copied

I knew REPLACE was wrong. It overwrites changes on the new side with old values.

E was the surprise. Rails reset_session deletes the old session ID from the store and hands out a new one (Session#destroy in actionpack). A key deleted on v2 looks "missing" to EXISTS, so Phase 2 fetches it again from the old side. The user's own cookie already holds the new ID, so they never notice. But a session ID that logout should have killed is alive again until its TTL runs out. If that cookie had leaked, a door that should be shut is open again.

The fix was simple. Keep the list of keys copied in Phase 1, and have Phase 2 only look at keys not on that list. That's the last row.

C loses something under every rule. If the same session changes on both sides during the deploy, you can't keep both. Keeping that window short was the only answer we found.

DUMP only goes one way

A DUMP payload carries the RDB format version (the first 2 of the last 10 bytes). In the same experiment 6.2.6 wrote 9 and 7.2.17 wrote 11.

6.2.6 → 7.2.17   RESTORE OK
7.2.17 → 6.2.6   ERR DUMP payload version or checksum are wrong

If the receiving side is older than the sender, it fails. So this method can't carry data back in the rollback direction. Rollback means pointing the app back at the old cluster, and you should decide up front that sessions created only on v2 in the meantime are dropped. Our real v2 was ElastiCache 7.1. I didn't try the reverse direction on 7.1 itself. What I checked was local OSS 7.2.

You can't "wait until the Sidekiq queue is empty"

The first runbook said "quiet Sidekiq and wait until the queue drains". That's wrong. In the Sidekiq source (launcher.rb, manager.rb), quiet stops worker threads from fetching new jobs. Quiet it and the queue stops shrinking. What you need to stop is the inflow.

On top of that, Sidekiq pushes in-progress jobs back onto the queue when it shuts down (bulk_requeue in fetch.rb). Jobs can appear on the old queue at the very moment the last old task dies. So instead of waiting for it to drain, we waited until every old task was gone and then moved the leftover jobs to v2. By pop-push, not by copying. A copy runs the same job twice if an old worker is still alive. I didn't reproduce this part locally. I checked it in the source.

The endpoint name changes

Straight from the AWS docs:

no in-transit encryption   redis-01.7abc2d.0001.usw2.cache.amazonaws.com
in-transit encryption      master.ncit.ameaqx.use1.cache.amazonaws.com

The node number like 0001 disappears and master. is prefixed. The read side is replica.. Build the hostname by hand and you'll get it wrong.

The reason is the TLS certificate. The TLS client example in the AWS docs verifies the host against *.mycluster.fnjyzo.use1.cache.amazonaws.com. Every name sits under the cluster name so one wildcard certificate covers them. A wildcard matches exactly one label. I made a certificate of the same shape locally and connected to a TLS Redis.

cert  *.demo-rg-v2.abc123.use1.cache.amazonaws.com
OK    master.demo-rg-v2.abc123.use1.cache.amazonaws.com
OK    replica.demo-rg-v2.abc123.use1.cache.amazonaws.com
FAIL  demo-rg-v2.abc123.ng.0001.use1.cache.amazonaws.com   (old shape)
FAIL  master.demo-rg.abc123.use1.cache.amazonaws.com       (forgot -v2)

Keep the old shape and just swap the name, and the TCP connection works but certificate verification fails. Feeding terraform's primary_endpoint_address output straight into the app config is the safe way. The scheme is rediss://, not redis://.

Real ElastiCache isn't reachable from outside the VPC, so in production we ran a one-off probe task on ECS and connected from there. One more thing: with in-transit encryption, a certificate containing the cluster name goes into the public Certificate Transparency logs (AWS docs). Don't put a customer name in the cluster name.

If I did it again, in this order

  • Build v2 under a different name with both at-rest and in-transit on. Upgrade the engine version here too

  • Take the Redis address in app config from the primary_endpoint_address output. rediss://

  • Phase 1 just before the deploy, Phase 2 right after. Phase 2 without REPLACE, only keys that are missing on v2 and not on the Phase 1 list

  • For queues, stop the inflow, or pop-push after every old task is gone

  • Rollback = point the app back. Write down in the plan that data can't go back

If you need to switch Redis or a database on AWS without stopping the service, ask via https://www.negabaro.com/contact. We also plan and rehearse migrations like this together.

How It Works

1

At-rest encryption is set only at creation; build a new cluster alongside and switch the app over

2

With Rack expire_after, the remaining TTL is the last visit; moving only the last 7 days is enough

3

Phase 2: no REPLACE, only keys missing on v2 and not copied in Phase 1; EXISTS alone revives logged-out sessions

4

DUMP goes older to newer only (6.2.6 RDB 9 to 7.2 RDB 11 works, not the reverse); rollback only repoints the app

5

With in-transit encryption the endpoint is master.<name>.<token>…; use primary_endpoint_address as is, with rediss://

Use Cases

A security review flags ElastiCache encryption and you have to add it to an existing cluster Moving a Rails app with sessions and Sidekiq queues in Redis to a new Redis without downtime After enabling TLS, the Redis connection fails certificate verification
← terraform apply Succeeded but AWS Did Not Change — ElastiCache Pending Modifications and the "No changes" Trap Since Provider 6.58