ECS Exec Session Logs Are Empty? Check Your Image for script(1) First

Nothing we did inside production containers through ECS Exec was being recorded anywhere. After reading this you will know where to look first when session logs are empty (does the image have script?), what the cluster logging setting actually controls, and how to fix it with one line on Alpine.

ECS Exec opens a shell in a running container with aws ecs execute-command. An SSM agent runs inside the container and relays the session.

We knew who connected, not what they did

CloudTrail records the ExecuteCommand call, so you know who connected to which task and when. What happened after that, say, what someone looked up or changed in rails console, left no trace.

You do not miss it day to day. You miss it when a data leak is suspected and someone asks how many records were viewed. The only honest answer was "we don't know."

Wrong turn one: the setting was in the container definition

The terraform did have executeCommandConfiguration. It was inside the task definition's container_definitions.

container_definitions = jsonencode([{
  name = "app"
  executeCommandConfiguration = {   # there is no such field here
    logging = "DEFAULT"
  }
}])

This setting belongs to the cluster. In the ECS API, ExecuteCommandConfiguration is part of the cluster configuration.

resource "aws_ecs_cluster" "app" {
  configuration {
    execute_command_configuration {
      logging = "OVERRIDE"
      log_configuration { ... }
    }
  }
}

When we ran aws ecs describe-task-definition, the key from the container definition was simply gone. Dropped without an error. Since container_definitions is a JSON string, nothing complained at terraform validate, plan, or apply. Anyone reading the code would assume it was configured.

So we put it on the cluster where it belongs. Still no logs.

Wrong turn two: chasing the loudest error

The agent log inside the container (/var/log/amazon/ssm/amazon-ssm-agent.log) had lines like these (shortened).

ERROR [DataBackend] unable to generate log data: Failed to generate transcript with the following errors: ...
WARN  [DataBackend] Signed HeadBucket request failed, continuing to fallback logic
ERROR [DataBackend] ... InvalidBucketName: The specified bucket is not valid.

The bucket name showed up as a literal {{s3BucketName}}, so we spent a long time suspecting the S3 settings, IAM, and KMS. All three were fine.

The cause was the first line, the one saying the transcript could not be generated. The bucket error came after it as a side effect. Read logs in time order, starting from the first error. The loudest error is not always the culprit.

The real cause: no /usr/bin/script in the image

The cluster ran two containers. The Alpine-based app container had zero session logs. A Debian-based container for another service had full session logs, without anyone touching its settings. Same settings, different results. That leaves the image.

The SSM agent source (agent/session/shell/shell_unix.go) explains it. When a session ends, the agent runs this:

script <log file> -c "cat <file holding the session I/O>"

script(1) records terminal I/O to a file. If that fails, the agent tries once more without -c. If that fails too, it reports Failed to generate transcript and returns before reaching the code that uploads to S3 or CloudWatch. With no binary, both attempts fail.

The ECS docs say it in one sentence: the container image needs script and cat installed for command logs to be uploaded to S3 or CloudWatch Logs. We had read past it.

The current agent source returns right there when the transcript fails. The bucket errors that followed in our log probably came from a different agent version in the container, but we did not track it down version by version.

The logging setting only picks the destination

The cluster logging setting has three values (docs).

• NONE: send nothing

• DEFAULT: send to the task definition's awslogs log group, the same place as the app logs

• OVERRIDE: send to the CloudWatch log group or S3 bucket in log_configuration

script sits one step earlier. It decides whether there is a file to send at all. If that step is blocked, changing the later one gets you zero logs every time. We tried.

The last row is the point. Add script and session logs start landing in the app log group with the default setting, without touching the cluster at all.

Only Alpine lacks it: we checked

We ran the official images in Docker (2026-11-09).

$ docker run --rm alpine:3.20 ls -l /usr/bin/script
ls: /usr/bin/script: No such file or directory

$ docker run --rm debian:bookworm-slim sh -c 'ls -l /usr/bin/script; dpkg -S /usr/bin/script'
-rwxr-xr-x 1 root root 133288 Nov 21  2024 /usr/bin/script
bsdutils: /usr/bin/script

Neither Alpine 3.16 nor 3.20 has it, and busybox has no script applet either. On Debian, bsdutils is an essential package, so even slim images ship with it. That is why this trap only shows up on Alpine-based images, and without a single warning. Alpine variants of official language images such as ruby:*-alpine share the same base.

The fix: one line

RUN apk add --no-cache util-linux-misc

On Alpine 3.16 this installs util-linux-misc-2.38-r1 (4,148 KiB installed). On 3.20 it is 2.40.1-r1 (6,200 KiB). Package splits can change between releases, so check which package owns the file afterwards with apk info -W /usr/bin/script.

Run the same command the agent uses and you get this:

$ script -q -c "cat /tmp/ipc.log" /tmp/out.txt < /dev/null
$ cat /tmp/out.txt
Script started on 2026-11-09 10:54:36+00:00 [COMMAND="cat /tmp/ipc.log" <not executed on terminal>]
id
uid=0(root)

Script done on 2026-11-09 10:54:36+00:00 [COMMAND_EXIT_CODE="0"]

Session logs in CloudWatch have exactly this Script started on ... / Script done on ... shape. The first time we saw that header we wondered where it came from. It is just the output format of script(1).

Same symptom? Go in this order

1. Is the config on the cluster? aws ecs describe-clusters --clusters <cluster> --include CONFIGURATIONS. Anything in the task definition is ignored
2. Does the container have script? aws ecs execute-command ... --command "ls -l /usr/bin/script". This is usually the answer
3. Look at the first ERROR in the agent log. If Failed to generate transcript comes first, you can ignore the S3 errors after it

Other things we stepped on

• Cluster settings only reached newly started tasks. Running tasks kept the old behavior. To test, replace the tasks with aws ecs update-service --force-new-deployment. Not knowing this cost us another round

• --configuration '{}' did not clear it. We removed the block in terraform to roll back, and the AWS side kept the old config. Passing the key explicitly, '{"executeCommandConfiguration":{}}', cleared it

• Do not trust storedBytes. For CloudWatch Logs log streams, storedBytes has always been 0 since 2019-06-17 (API docs). Zero does not mean empty. Read with get-log-events

• A stream existing does not mean logs exist. We had more than 30 ecs-execute-command-* streams. Every one was empty

• Retention. DEFAULT inherits the app log group's retention. Seven days is useless for an incident found weeks later. Session output carries connection strings and tokens in plain text, so move audit logs to their own group, keep them longer, and narrow who can read them

One more thing to check when you pick a small image

We chose Alpine to keep the image small. That choice quietly switched off our audit log for more than two years. Image size comparisons will never show you this.

terraform that cannot validate a JSON string, an API that drops unknown keys, an agent that stays quiet when a binary is missing. Each design makes sense on its own. Together, nobody noticed. Features you only need on a bad day, like audit logs, deserve one deliberate look.

If you want help sorting out audit logging on ECS or Fargate, or settings where terraform and reality have drifted apart, reach out at https://www.negabaro.com/contact.