Module

orchestration.streaming

Streaming build orchestration for memory-optimized builds.

Uses knowledge graph analysis to process pages in optimal order for memory efficiency. Hub-first strategy: Keep highly connected pages in memory, stream leaves.

View source
1 Class

Classes

StreamingRenderOrchestrator
Memory-optimized page rendering using knowledge graph analysis. Strategy: 1. Build knowledge graph…
3

Memory-optimized page rendering using knowledge graph analysis.

Strategy:

  1. Build knowledge graph to identify connectivity
  2. Process hubs first (keep in memory - they're needed often)
  3. Stream leaves in batches (release immediately after rendering)
  4. Result: 80-90% memory reduction for large sites

Best for: Sites with 5K+ pages

Methods 1

process
Render pages in memory-optimized batches using connectivity analysis. Uses bui…
10 None 
def process(self, pages: list[Page], parallel: bool = True, quiet: bool = False, tracker: DependencyTracker | None = None, stats: BuildStats | None = None, batch_size: int = 100, progress_manager: Any | None = None, reporter: Any | None = None, build_context: Any | None = None, changed_sources: set[Path] | None = None) -> None

Render pages in memory-optimized batches using connectivity analysis.

Uses build_context.knowledge_graph if available to avoid rebuilding the graph multiple times per build.

Parameters 10
pages list[Page]

List of pages to render
parallel bool

Whether to use parallel rendering
quiet bool

Whether to suppress progress output (minimal output mode)
tracker DependencyTracker | None

Dependency tracker for incremental builds
stats BuildStats | None

Build statistics tracker
batch_size int

Number of leaves to process per batch
progress_manager Any | None

Optional progress manager to use for unified progress display
reporter Any | None

Optional progress reporter for output
build_context Any | None

Optional BuildContext with cached knowledge graph
changed_sources set[Path] | None
Internal Methods 2
__init__
Initialize streaming render orchestrator.
1 None 
def __init__(self, site: Site)

Initialize streaming render orchestrator.

Parameters 1
site Site

Site instance containing pages
_render_batches
Render pages in batches with optional memory release.
11 None 
def _render_batches(self, renderer: RenderOrchestrator, pages: list[Page], batch_size: int, parallel: bool, quiet: bool, tracker: DependencyTracker | None, stats: BuildStats | None, batch_label: str = 'pages', release_memory: bool = False, progress_manager: Any | None = None, changed_sources: set[Path] | None = None) -> None

Render pages in batches with optional memory release.

Parameters 11
renderer RenderOrchestrator

RenderOrchestrator instance
pages list[Page]

Pages to render
batch_size int

Pages per batch
parallel bool

Use parallel rendering
quiet bool

Whether to suppress progress output
tracker DependencyTracker | None

Dependency tracker
stats BuildStats | None

Build statistics
batch_label str

Label for logging
release_memory bool

Whether to force garbage collection after each batch
progress_manager Any | None

Optional progress manager to use for unified progress display
changed_sources set[Path] | None