Graceful shutdown and connection draining, reproduced

A graceful shutdown should be boring: on SIGTERM, the process stops accepting new work, finishes what is already in flight, then exits. Most processes do the opposite — they die the instant the signal lands, and every request they were holding dies with them. That gap is where a routine deploy becomes a page full of 502 Bad Gateway. I reproduced it with a throwaway nginx in front of a tiny app that takes one second per request, sent SIGTERM mid-flight, and watched connection draining either happen or not — the mechanic a real zero-downtime deploy depends on.

The setup was a throwaway nginx:1.27.5 proxying python:3.14 on a Docker bridge network, all on 127.0.0.1, no privileges. Everything below is real output.

What SIGTERM should do — and what it usually does

SIGTERM is the polite signal: unlike SIGKILL it is catchable, and the intended contract is three steps:

1. stop accepting new connections (stop reading the listening socket),
2. finish the requests already in flight,
3. exit.

Most programs do step 3 and only step 3. The default disposition of SIGTERM is “terminate immediately”, so unless the app installs a handler, requests are still open when the process dies — the client sees a reset, or a 502 if a proxy is in front.

The shutdown looks like this:

One trap first: PID 1 ignores signals it has no handler for. My first run sent SIGTERM and nothing happened — the app kept serving, because Python was PID 1. That is why containers ship a tiny init (docker run --init, Compose init: true) that forwards signals to a child that can handle them.

Why a deploy causes 502s

The outage is a race between two independent decisions:

• the load balancer / reverse proxy decides when to stop routing to an instance, and
• the app decides when to stop accepting connections.

If the app stops accepting before the proxy stops sending, the proxy keeps dialing a socket nobody is listening on. Every request in that window fails. The app did nothing wrong; the two sides just disagreed about the order. The whole craft of connection draining is making the proxy go first and giving the app a grace window to finish.

Under load, that tiny window is exactly where your slowest requests live.

The failure: SIGTERM with no handler

The app sleeps one second per request and, in the “no drain” build, installs no signal handler. I fire six concurrent requests through nginx and send SIGTERM at 0.4s:

$ ./run-case.sh 6 0.4
>>> sending SIGTERM to app container
HTTP 502  0.597s  req=4
HTTP 502  0.601s  req=3
HTTP 502  0.601s  req=2
HTTP 502  0.604s  req=6
HTTP 502  0.604s  req=5
HTTP 502  0.611s  req=1

All six fail, and they fail early — around 0.6s instead of the full 1.0s. The requests had a second of work left; the process just vanished underneath them. The app log tells the same story from the inside: six START lines and not a single FINISH.

START  req=1 active=1
START  req=5 active=2
START  req=6 active=3
START  req=2 active=4
START  req=3 active=5
START  req=4 active=6
# process terminated — no FINISH lines ever print

nginx names the cause exactly, in its error log:

[error] upstream prematurely closed connection while reading response
        header from upstream, upstream: "http://172.22.0.2:8080/"

“Prematurely closed” is nginx’s way of saying the upstream TCP connection went away before a response header arrived. The client gets 502.

The fix: drain in-flight, then exit

Now the app handles the signal. On SIGTERM it stops accepting, waits for the in-flight counter to reach zero, and exits:

def on_term(signum, frame):
    log("signal received: stop accepting, drain in-flight")
    threading.Thread(target=server.shutdown, daemon=True).start()

signal.signal(signal.SIGTERM, on_term)   # only installed when draining is on

server.serve_forever()                   # returns once shutdown() stops the accept loop
while active_requests > 0:               # work already accepted keeps running
    time.sleep(0.05)
sys.exit(0)

Same six requests, same mid-flight signal, one difference in behavior:

$ DRAIN=1 ./run-case.sh 6 0.4
>>> sending SIGTERM to app container
HTTP 200  1.001s  req=5
HTTP 200  1.001s  req=1
HTTP 200  1.001s  req=6
HTTP 200  1.001s  req=3
HTTP 200  1.001s  req=4
HTTP 200  1.001s  req=2

And the log, now with the shutdown visible:

signal 15 received: stop accepting, drain in-flight
accept loop stopped; waiting for in-flight requests
FINISH req=1 elapsed=1.00s
FINISH req=4 elapsed=1.00s
...
all in-flight done; exiting cleanly

Six 200s instead of six 502s — most of what “we handle SIGTERM properly” means.

New requests during the drain still fail

Draining does not make the instance invincible, and this is the part people skip: while it finishes old work, it is not accepting new connections. Anything the proxy sends now fails:

IN-FLIGHT req=A  200  3.00s     # accepted before SIGTERM, finishes
IN-FLIGHT req=B  200  3.04s
NEW       req=C  502  1.96s     # proxy sent a new connection into a closed listener

So draining alone is not enough — pair it with the proxy stopping traffic. The correct order is: tell the load balancer to stop sending → wait a beat → send SIGTERM → let the app drain. Kubernetes models that beat, and so should you.

What nginx does when an upstream disappears

nginx makes this explicit. With max_fails=0 it never marks the backend “down”, so you see the truth instead of a cached failure. Knobs that matter:

• proxy_next_upstream off; — don’t silently retry a different backend, which hides a broken deploy behind a slow 200.
• proxy_connect_timeout 1s; — bound how long a client waits for a socket that is no longer listening (I saw a 504 when the port refused instead of reset).
• keepalive N; with proxy_http_version 1.1; and proxy_set_header Connection ""; — reuse upstream connections, which is faster but means the proxy must also retire keepalive connections when an upstream leaves.

nginx also has two very different ways to go down:

nginx -s reload   →  HTTP 200 ×4   workers replaced, in-flight requests never interrupted
nginx -s quit     →  HTTP 200 ×4   graceful shutdown: drain, then exit
nginx -s stop     →  HTTP 000 ×4   fast shutdown: connections cut mid-response

reload is the zero-downtime primitive: the master reads new config, starts fresh workers, and lets old ones finish. quit is graceful shutdown. stop yanks the cable — clients get nothing (000 is curl reporting a connection that died with no HTTP response at all). Most people who think they want “restart” actually want reload; I go deeper in the nginx config cookbook.

Kubernetes: preStop and terminationGracePeriodSeconds

The orchestrator cannot fix a broken shutdown, but it gives you two handles. When a pod is deleted, Kubernetes removes it from Service endpoints and starts the termination countdown at the same time — so a preStop hook delays the app, letting endpoint removal reach kube-proxy and the load balancer first. terminationGracePeriodSeconds bounds the whole thing; when it expires, SIGKILL follows.

terminationGracePeriodSeconds: 30   # budget: must exceed longest in-flight request + queue
lifecycle:
  preStop:
    exec: { command: ["sh", "-c", "sleep 5"] }   # let deregistration reach the proxies

The distinction that trips people up: a readiness probe controls new traffic; a liveness probe decides whether a stuck process gets restarted. A draining app should fail readiness while staying alive — never fail liveness during shutdown, or the kubelet kills the requests you were saving. That “announce unready, then drain” handshake is the core of my zero-downtime deploy setup.

A per-stack graceful-shutdown checklist

Everything above is the same three steps with different spellings. Match the timeout to your slowest request, and on every stack make sure the process actually receives SIGTERM:

# systemd
[Service]
KillSignal=SIGTERM      # the polite signal
KillMode=mixed          # signal the main process, then the whole cgroup
TimeoutStopSec=30       # then SIGKILL; must exceed longest request + queue

# Dockerfile
STOPSIGNAL SIGTERM      # what `docker stop` sends; default is already SIGTERM, be explicit

# compose.yaml
services:
  app:
    init: true                    # PID 1 signal forwarding + zombie reaping
    stop_grace_period: 30s        # docker stop's patience before SIGKILL

The checklist, compressed: (1) install a SIGTERM handler that stops accepting but finishes in-flight work; (2) make sure the signal reaches the process — exec in entrypoints, init: true for PID 1; (3) stop routing before you stop accepting (nginx upstream weight, Kubernetes preStop, or a deregistration delay); (4) set the stop timeout longer than your slowest request; (5) prefer reload over restart, and drain over both.

None of this is exotic: one handler, one counter, one honest handshake with the proxy. The 502s were never the app failing — just the app and the load balancer disagreeing about who leaves the room first. Get the order right and the deploy is invisible, which is exactly what you want.