Graceful Shutdown for Queue Workers on ECS

Deploying a web server is comparatively forgiving: stop accepting new requests, finish the small number already in flight, and exit. Queue workers have a harder problem. A job may run for minutes, modify external systems, and be delivered again if its acknowledgement never reaches the broker.

Graceful shutdown is therefore part of job correctness, not just deployment polish.

Understand the termination sequence

When ECS replaces a task, the container receives a termination signal before it is forcibly stopped. The application has a bounded window to react.

A worker should respond in this order:

1. Mark itself as draining.
2. Stop fetching new jobs.
3. Allow in-flight work to finish within the remaining deadline.
4. Acknowledge completed work.
5. Release resources and exit successfully.

The signal handler should initiate shutdown, not perform all cleanup synchronously. The main worker loop needs to observe the draining state and coordinate active jobs.

Align the time budgets

Several timeouts must agree:

• the container stop timeout;
• the queue visibility timeout or lease duration;
• the application's shutdown deadline;
• the maximum expected job duration.

If a job can run longer than its visibility timeout, another worker may receive it while the first is still processing. Extend the lease while work progresses or split the job into smaller resumable steps.

The application shutdown deadline should be shorter than the platform deadline, leaving time for connections and telemetry to flush.

Assume duplicate delivery

Even a perfect shutdown path cannot guarantee exactly-once execution. A process can fail after an external side effect succeeds but before acknowledgement.

Make effects idempotent with techniques such as:

• a stable job or operation identifier;
• a database uniqueness constraint;
• conditional state transitions;
• an idempotency key for external APIs;
• an inbox table recording processed messages.

Retries then become a normal recovery mechanism instead of a source of duplicated emails, payments, or records.

Make draining observable

Track active jobs, shutdown duration, forced terminations, retry counts, lease extensions, and the age of the oldest queued message. Logs should carry the job identifier and deployment or task identifier so an interrupted execution can be followed across attempts.

A deployment that completes while forced terminations rise is not healthy. Without these signals, the problem often appears later as a mysterious duplicate effect.

The operational contract

Reliable workers need a clear contract: what can be retried, which effects are idempotent, how long work may take, and what happens when the process is asked to stop. When that contract is designed before deployment, rolling releases become routine rather than a gamble.