# supervisor URL: /pounce/api/supervisor/ Section: api Description: Supervisor — spawns, monitors, and restarts workers. The supervisor sits between the ``Server`` and the ``Worker`` layer. It detects the GIL state at startup and spawns workers as either threads (on nogil / 3.14t) or processes (on GIL builds). The worker implementation is identical in both modes — only the spawning mechanism differs. Responsibilities: - Spawn N workers with their sockets - Monitor worker health (is_alive check on a watchdog loop) - Restart crashed workers (up to ``max_restarts`` per window) - Coordinate graceful shutdown via ``threading.Event`` - Forward SIGINT/SIGTERM to workers --- > For a complete page index, fetch /pounce/llms.txt. Open LLM text (/pounce/api/supervisor/index.txt) Share with AI Ask Claude (https://claude.ai/new?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpounce%2Fapi%2Fsupervisor%2Findex.txt) Ask ChatGPT (https://chatgpt.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpounce%2Fapi%2Fsupervisor%2Findex.txt) Ask Gemini (https://gemini.google.com/app?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpounce%2Fapi%2Fsupervisor%2Findex.txt) Ask Copilot (https://copilot.microsoft.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpounce%2Fapi%2Fsupervisor%2Findex.txt) Module # `supervisor` Supervisor — spawns, monitors, and restarts workers. The supervisor sits between the`Server` and the `Worker`layer. It detects the GIL state at startup and spawns workers as either threads (on nogil / 3.14t) or processes (on GIL builds). The worker implementation is identical in both modes — only the spawning mechanism differs. Responsibilities: - Spawn N workers with their sockets - Monitor worker health (is_alive check on a watchdog loop) - Restart crashed workers (up to`max_restarts`per window) - Coordinate graceful shutdown via`threading.Event` - Forward SIGINT/SIGTERM to workers 4Classes5Functions ## Classes `TCPWorker` 4 ▼ Structural contract for TCP workers (Worker and SyncWorker). Both async and sync workers implement… Bases: `Protocol` Structural contract for TCP workers (Worker and SyncWorker). Both async and sync workers implement this interface. The supervisor uses it for lifecycle management: spawning, draining, and idle checks. #### Methods `run` 0 ▼ `def run(self) -> None` `set_lifespan_state` 1 ▼ `def set_lifespan_state(self, state: dict[str, Any]) -> None` ##### Parameters Name Type Description `state` `—` `start_draining` 0 ▼ `def start_draining(self) -> None` `is_idle` 0 `bool` ▼ `def is_idle(self) -> bool` ##### Returns `bool` `_WorkerHandle` 1 ▼ Metadata about a running worker (thread or process). Metadata about a running worker (thread or process). #### Methods Internal Methods 1 ▼ `__init__` 4 ▼ `def __init__(self, worker_id: int, target: threading.Thread | multiprocessing.Process, worker: TCPWorker | None, generation: int = 0) -> None` ##### Parameters Name Type Description `worker_id` `—` `target` `—` `worker` `—` `generation` `—` Default:`0` `_H3WorkerHandle` 1 ▼ Metadata about a running H3 worker (UDP/QUIC). Metadata about a running H3 worker (UDP/QUIC). #### Methods Internal Methods 1 ▼ `__init__` 4 ▼ `def __init__(self, worker_id: int, target: threading.Thread | multiprocessing.Process, worker: H3Worker | None = None, reload_shutdown_event: threading.Event | None = None) -> None` ##### Parameters Name Type Description `worker_id` `—` `target` `—` `worker` `—` Default:`None` `reload_shutdown_event` `—` Default:`None` `Supervisor` 29 ▼ Spawn and supervise N workers as threads or processes. The supervisor detects the GIL state and pi… Spawn and supervise N workers as threads or processes. The supervisor detects the GIL state and picks the appropriate spawning strategy automatically. Workers share the frozen `ServerConfig` (/pounce/api/config/#ServerConfig) and (in thread mode) the ASGI app reference. #### Methods `mode` 0 `WorkerMode (/pounce/api/_runtime/#WorkerMode)` ▼ The active worker mode (``"thread"``, ``"process"``, or ``"subinterpreter"``). property `def mode(self) -> WorkerMode` The active worker mode (`"thread"`, `"process"`, or `"subinterpreter"`). ##### Returns `WorkerMode (/pounce/api/_runtime/#WorkerMode)` `worker_count` 0 `int` ▼ Number of workers the supervisor manages. property `def worker_count(self) -> int` ##### Returns `int` `set_lifespan_state` 1 ▼ Set the lifespan state dict to be shared with all workers. `def set_lifespan_state(self, state: dict[str, Any]) -> None` ##### Parameters Name Type Description `state` `—` The state dict populated during lifespan startup. `run` 2 ▼ Start all workers and block until shutdown. Installs signal handlers, spawns w… `def run(self, sockets: list[socket.socket], udp_sockets: list[socket.socket] | None = None) -> None` Start all workers and block until shutdown. Installs signal handlers, spawns workers, runs the health-check loop, then joins all workers on shutdown. ##### Parameters Name Type Description `sockets` `—` One TCP socket per worker, created by`create_listeners()`. `udp_sockets` `—` Optional UDP sockets for HTTP/3 workers. Default:`None` `shutdown` 0 ▼ Signal all workers to stop (non-blocking). `def shutdown(self) -> None` `restart_workers` 0 ▼ Gracefully restart all workers (for dev reload). Signals all running workers t… `def restart_workers(self) -> None` Gracefully restart all workers (for dev reload). Signals all running workers to stop, waits for them to drain, clears the shutdown event, and spawns fresh workers. Serialized with graceful_reload and watch-loop respawns via _lifecycle_lock. Skips if a reload is already in progress. When an`app_path`was provided and workers run as threads, the app module is reimported so that code changes on disk take effect. Process-based workers inherit the parent via fork and are fully replaced on restart, so reimport is not needed. `graceful_reload` 0 ▼ Perform zero-downtime rolling restart of all workers. This method implements a… `def graceful_reload(self) -> None` Perform zero-downtime rolling restart of all workers. This method implements a rolling restart strategy: - Reimport the app (thread mode only) - Spawn new worker generation - Mark old workers for draining (finish existing, reject new connections) - Wait for old workers to become idle - Shut down old workers This ensures zero dropped requests during code reload. Note: Only works in thread mode. In process mode, falls back to restart_workers() which has brief downtime. Internal Methods 22 ▼ `__init__` 7 ▼ `def __init__(self, config: ServerConfig, app: ASGIApp, *, mode: WorkerMode | None = None, ssl_context: ssl.SSLContext | None = None, lifecycle_collector: LifecycleCollector | None = None, app_path: str | None = None, sync_app: SyncApp | None = None) -> None` ##### Parameters Name Type Description `config` `—` `app` `—` `mode` `—` Default:`None` `ssl_context` `—` Default:`None` `lifecycle_collector` `—` Default:`None` `app_path` `—` Default:`None` `sync_app` `—` Default:`None` `_await_initial_workers_ready` 0 ▼ Wait for every initial worker hook and listener to succeed. `def _await_initial_workers_ready(self) -> None` `_raise_startup_failure` 2 `Never` ▼ Abort initial startup with the stable operator-facing error. `def _raise_startup_failure(self, message: str, hint: str) -> Never` ##### Parameters Name Type Description `message` `—` `hint` `—` ##### Returns `Never` `_wait_for_subinterpreter_generation_ready` 1 `str | None` ▼ Wait for a replacement generation without changing global state. Initial start… `def _wait_for_subinterpreter_generation_ready(self, iic_queues: list[tuple[Any, Any]]) -> str | None` Wait for a replacement generation without changing global state. Initial startup failures abort the server through `_raise_startup_failure` (/pounce/api/supervisor/#Supervisor)(). A reload is different: the old generation is still healthy, so a replacement failure must be reported to the caller without setting the supervisor shutdown event. ##### Parameters Name Type Description `iic_queues` `—` ##### Returns `str | None` `_signal_workers_start_draining` 0 ▼ Mark async workers as draining (503 new connections) during shutdown. Thread-m… `def _signal_workers_start_draining(self) -> None` Mark async workers as draining (503 new connections) during shutdown. Thread-mode workers expose a `Worker` (/pounce/api/worker/#Worker) / `SyncWorker` (/pounce/api/sync_worker/#SyncWorker) instance; process workers do not (`handle.worker` (/pounce/api/worker/) is None). Subinterpreter workers receive drain commands via IIC queue. `_restart_workers_impl` 0 ▼ Internal implementation of restart_workers (no lock). `def _restart_workers_impl(self) -> None` `_graceful_reload_impl` 0 ▼ Internal implementation of graceful_reload (no lock). `def _graceful_reload_impl(self) -> None` `_uses_sync_infrastructure` 0 `bool` ▼ True when AsyncPool/AcceptDistributor apply (thread mode, sync exec). `def _uses_sync_infrastructure(self) -> bool` ##### Returns `bool` `_build_async_pool` 0 `tuple[AsyncPool (/pounce/api/async_pool/#AsyncPool) | None, …` ▼ Build and start a fresh AsyncPool bound to the current ``self._app``. Returns … `def _build_async_pool(self) -> tuple[AsyncPool | None, threading.Thread | None]` Build and start a fresh AsyncPool bound to the current`self._app`. Returns the pool and its thread handle without mutating`self`so the caller can repoint`self._async_pool`and retire the old one in the order it chooses (issue #102). ##### Returns `tuple[AsyncPool (/pounce/api/async_pool/#AsyncPool) | None, threading.Thread | None]` `_build_accept_distributor` 0 `tuple[queue.Queue[tuple[…` ▼ Build and start a fresh AcceptDistributor for shared-socket sync mode. Returns… `def _build_accept_distributor(self) -> tuple[queue.Queue[tuple[socket.socket, object]] | None, threading.Thread | None, threading.Event | None]` Build and start a fresh AcceptDistributor for shared-socket sync mode. Returns`(conn_queue, handle, drain_event)` (all `None`when a distributor does not apply) without mutating`self`. ##### Returns `tuple[queue.Queue[tuple[socket.socket, object]] | None, threading.Thread | None, threading.Event | None]` `_setup_sync_infrastructure` 0 ▼ Create AsyncPool and AcceptDistributor for sync worker mode. `def _setup_sync_infrastructure(self) -> None` `_create_worker` 2 `Worker (/pounce/api/worker/#Worker) | SyncWorker (/pounce/api/sync_worker/#SyncWorker)` ▼ Create a Worker or SyncWorker based on the execution mode. `def _create_worker(self, worker_id: int, socket_index: int) -> Worker | SyncWorker` ##### Parameters Name Type Description `worker_id` `—` `socket_index` `—` ##### Returns `Worker (/pounce/api/worker/#Worker) | SyncWorker (/pounce/api/sync_worker/#SyncWorker)` `_worker_socket` 1 `socket.socket` ▼ Return a worker-owned socket handle. Thread workers always receive a duplicate… `def _worker_socket(self, socket_index: int) -> socket.socket` Return a worker-owned socket handle. Thread workers always receive a duplicated handle so one generation can close its asyncio server without invalidating the canonical listener kept in`self._sockets`for the next reload generation. This applies both when workers share one listener object (macOS / Windows fallback) and when each worker has its own`SO_REUSEPORT` socket (Linux): without a dup, an old worker closing its asyncio server would close the supervisor-owned socket and leave reload with`EBADF`. ##### Parameters Name Type Description `socket_index` `—` ##### Returns `socket.socket` `_spawn_worker` 1 ▼ Create and start a single worker. `def _spawn_worker(self, worker_id: int) -> None` ##### Parameters Name Type Description `worker_id` `—` `_spawn_subinterpreter_worker` 1 ▼ Create and start a worker inside a subinterpreter (PEP 734). Each worker runs … `def _spawn_subinterpreter_worker(self, worker_id: int) -> None` Create and start a worker inside a subinterpreter (PEP 734). Each worker runs in a dedicated thread that hosts a subinterpreter. Communication uses IIC queues (tagged tuples) instead of direct method calls or threading.Events. The supervisor passes config as JSON, the app as an import path, and the socket as a dup'd file descriptor — all IIC-safe types. ##### Parameters Name Type Description `worker_id` `—` `_spawn_h3_worker` 1 ▼ Create and start a single H3 (HTTP/3) worker. `def _spawn_h3_worker(self, worker_id: int) -> None` ##### Parameters Name Type Description `worker_id` `—` `_respawn_worker` 1 ▼ Restart a crashed worker if within restart budget. Serialized with restart_wor… `def _respawn_worker(self, worker_id: int) -> None` Restart a crashed worker if within restart budget. Serialized with restart_workers/graceful_reload via _lifecycle_lock. Skips if a reload is in progress (avoids overlapping restarts). ##### Parameters Name Type Description `worker_id` `—` `_watch` 0 ▼ Health-check loop — detects crashed workers and restarts them. `def _watch(self) -> None` `_drain` 0 ▼ Wait for all workers to finish draining connections, then clean up. Signals sh… `def _drain(self) -> None` Wait for all workers to finish draining connections, then clean up. Signals shutdown to all workers, waits for them to finish processing active connections (see`shutdown_timeout` on `ServerConfig` (/pounce/api/config/#ServerConfig)), then force-terminates process workers that haven't stopped. Workers will reject new connections but finish existing ones for clean shutdown (important for Kubernetes graceful termination). `_reclaim_subinterpreter_fd` 1 ▼ Close a subinterpreter worker's dup'd listener FD, if still recorded. Subinter… `def _reclaim_subinterpreter_fd(self, handle: _WorkerHandle) -> None` Close a subinterpreter worker's dup'd listener FD, if still recorded. Subinterpreter workers run as daemon threads the parent cannot force kill (see `_force_stop` (/pounce/api/supervisor/#Supervisor)()). When such a worker stops abnormally — a crash/respawn, or a force-stop of an old generation that never drained — its bootstrap finally-block may not have closed the dup'd listener FD. Each abnormal cycle would then leak one FD until the process exhausts its descriptor budget (issue #106). The parent recorded the FD on `sock_fd`, so it can reclaim it here. The generated bootstrap wrapper acknowledges its worker-side close with a parent-owned Event before the host thread exits. In that case this method only clears the record: calling `os.close` (/pounce/api/protocols/ws/#WSProtocol) on the old numeric value could close an unrelated resource if the OS already reused it. Without an acknowledgement, the parent still owns abnormal cleanup. ##### Parameters Name Type Description `handle` `—` `_force_stop` 2 ▼ Force-terminate a worker that did not drain in time. Process workers receive S… `def _force_stop(self, handle: _WorkerHandle, join_timeout: float) -> None` Force-terminate a worker that did not drain in time. Process workers receive SIGTERM then SIGKILL. Thread workers cannot be terminated from Python; they are daemon threads and may outlive this join. ##### Parameters Name Type Description `handle` `—` `join_timeout` `—` `_install_signals` 0 ▼ Install SIGINT/SIGTERM handlers to trigger graceful shutdown. Only effective w… `def _install_signals(self) -> None` Install SIGINT/SIGTERM handlers to trigger graceful shutdown. Only effective when the supervisor runs on the main thread (e.g., direct testing). In production the supervisor runs inside a `run_in_executor` thread, so `signal.signal()`will fail silently. `Server` (/pounce/api/server/#Server) installs asyncio signal handlers that call `supervisor.shutdown()`instead. ## Functions `_get_fork_context` 0 `multiprocessing.context.…` ▼ Return a ``"fork"`` multiprocessing context. Process workers must use ``fork``… `def _get_fork_context() -> multiprocessing.context.BaseContext` Return a`"fork"`multiprocessing context. Process workers must use`fork`so the ASGI app (which may contain closures from middleware wrappers or framework decorators) is inherited via the forked address space rather than pickled. The default start method on macOS and Windows is`spawn`, which pickles the target — any non-picklable object (closures, local functions, lambdas) in the app will crash the server at startup. ##### Returns `multiprocessing.context.BaseContext` `_parallel_join_targets` 2 `None` ▼ Join each worker thread/process in parallel with its own timeout. ``shutdown_t… `def _parallel_join_targets(targets: list[threading.Thread | multiprocessing.Process], timeout_per: float) -> None` Join each worker thread/process in parallel with its own timeout. `shutdown_timeout`is applied per worker (not split across N workers). Wall-clock time is roughly`timeout_per`when all workers finish together, instead of one shared deadline that starved later workers in the join order. ##### Parameters Name Type Description `targets` `list[threading.Thread | multiprocessing.Process]` Threads or processes to`join`. `timeout_per` `float` Maximum seconds to wait for each target. `_serialize_lifespan_state` 1 `str` ▼ Serialize IIC-safe lifespan state keys to JSON. Non-serializable values (DB po… `def _serialize_lifespan_state(state: dict[str, Any]) -> str` Serialize IIC-safe lifespan state keys to JSON. Non-serializable values (DB pools, HTTP clients, etc.) are skipped with a debug log. The resulting JSON string is passed to each subinterpreter worker via`prepare_main()`. ##### Parameters Name Type Description `state` `dict[str, Any]` ##### Returns `str` `_try_iic_get` 1 `tuple[Any, ...] | None` ▼ Non-blocking get from an IIC queue. Returns None if empty or unbound. `def _try_iic_get(queue: Any) -> tuple[Any, ...] | None` ##### Parameters Name Type Description `queue` `Any` ##### Returns `tuple[Any, ...] | None` `_target_id` 1 `str` ▼ Return an identifier string for a thread or process. `def _target_id(target: threading.Thread | multiprocessing.Process) -> str` ##### Parameters Name Type Description `target` `threading.Thread | multiprocessing.Process` ##### Returns `str`