Server Lifecycle

Graceful reload and shutdown with connection draining

4 min read 744 words

Pounce handles SIGHUP for graceful reload on supported multi-worker paths and SIGTERM / SIGINT for graceful shutdown. Both paths use connection draining: active requests get time to finish, while workers that are leaving service reject new connections.

Graceful Reload (SIGHUP)

On supported multi-worker thread and subinterpreter paths, send SIGHUP to perform a rolling restart with fresh code:

kill -HUP <pid>
# or with systemd:
systemctl reload pounce

What Happens

  1. Old workers continue handling existing requests
  2. App code is reimported and new workers spawn (generation N+1)
  3. Old workers enter drain mode (finish active requests, reject new ones)
  4. Once drained (or afterreload_timeout), old workers exit
Time 0s:   [Worker-0] [Worker-1] [Worker-2] [Worker-3]  (Gen 0)
           SIGHUP received
Time 0.1s: [Worker-0..3 draining] [Worker-4..7 accepting]  (Gen 0+1)
Time 5s:   [Worker-4] [Worker-5] [Worker-6] [Worker-7]  (Gen 1 only)

If the reimport fails, pounce logs the error and continues with the old code instead of swapping to the failed generation.

HTTP/3 uses a separate UDP/QUIC listener. Its proof ledger covers generation rotation, bounded drain, and orphan-thread cleanup. Under-budget streams finish; streams exceedingshutdown_timeoutare cancelled and QUIC closes.

Current subprocess proof covers SIGTERM mixed-traffic drain and SIGHUP recovery to serving traffic across the documented worker-mode matrix. The reproducible drain profile also drives SIGHUP followed by SIGTERM and records in-flight completion, bounded refusal outcomes, exit time, and orphan-worker absence. This mode-scoped proof does not imply lossless reload across every protocol; HTTP/3 keeps the bounded QUIC exception above rather than claiming TCP-identical semantics.

Configuration

config = ServerConfig(
    reload_timeout=60.0,  # Max drain time (default: 30s)
    workers=4,
)

systemd

[Service]
Type=notify
ExecStart=/usr/bin/pounce serve --app myapp:app --workers=4
ExecReload=/bin/kill -HUP $MAINPID

File Watching (Development)

For development, enable auto-reload on file changes:

config = ServerConfig(
    reload=True,
    reload_include=(".html", ".css"),  # Extra extensions
    reload_dirs=("templates",),        # Extra directories
)

Graceful Shutdown (SIGTERM)

On SIGTERM or SIGINT, pounce drains connections then exits:

  1. Emitspounce.worker.drainingwith worker/generation identity, reason, and timeout
  2. Stops accepting new connections immediately
  3. Finishes active requests (up toshutdown_timeout)
  4. Force-terminates work that exceeds the timeout
  5. Runs per-workerpounce.worker.shutdownhooks
  6. Completes ASGIlifespan.shutdown
  7. Releases listeners and exits with status 0
config = ServerConfig(
    shutdown_timeout=30.0,  # Per-worker drain time (default: 10s)
)

The draining hook is a Pounce ASGI extension scope. It is bounded to one second inside the existing shutdown budget. Stream registries can use (worker_id, generation)to emit a final SSE close event only to clients on the retiring generation; HTTP scopes carry the matching identity at scope["extensions"]["pounce.worker"]. Apps that do not recognize the hook may return or raise as they do for the existing worker hooks.

On graceful reload, old-generation streams receivereason="reload". They keep running until the app closes them or the configured drain deadline forces the old worker closed. New-generation streams use a different generation and are not part of that notification.

Kubernetes

spec:
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "sleep 5"]  # LB de-registration delay
    readinessProbe:
      httpGet:
        path: /readyz
        port: 8000
  terminationGracePeriodSeconds: 40  # > shutdown_timeout + preStop

Key: terminationGracePeriodSeconds must exceed shutdown_timeout+ preStop delay, or Kubernetes sends SIGKILL before drain completes.

Docker

Use exec form so signals reach pounce directly:

CMD ["pounce", "serve", "myapp:app", "--host", "0.0.0.0"]

systemd

[Service]
Type=notify
KillSignal=SIGTERM
KillMode=mixed
TimeoutStopSec=40s

Worker Mode Comparison

Thread Mode (3.14t) Process Mode (GIL) Subinterpreter Mode
Reload Rolling generation swap with old + new overlap Stop/start fallback may have a brief gap Replacement readiness, then old-generation acceptor retirement and bounded drain
Shutdown Drain per-thread Drain per-process IIC-coordinated bounded drain and shutdown
Scope Shared-interpreter ASGI workers Forked ASGI workers Stable isolated ASGI web workers; import path and compatible dependencies required

Thread mode requires Python 3.14t (free-threading). Process mode falls back to stop-all-then-start. Subinterpreter mode is explicit. Its lifecycle is stable for ASGI web workers, but dependency compatibility and JSON-safe lifespan-state limits still apply.

Troubleshooting

Workers not draining: Increasereload_timeout or shutdown_timeout. Check for long-lived connections (WebSocket, streaming).request_timeoutbounds request-body progress; application execution is bounded by lifecycle drain deadlines rather than this network-input setting.

SIGKILL before drain complete (Kubernetes): IncreaseterminationGracePeriodSeconds to exceed shutdown_timeout+ preStop delay.

Module reload failures: Pounce logs the import error and continues with the previous version.