Build Orchestration

Bengal's orchestration system coordinates builds through specialized orchestrators. TheSiteis a passive data container; orchestrators handle build logic.

Build Phases

The build executes 21 main phases (plus optional sub-phases like 1.5, 12.5, 19.5) organized into 5 groups:

Phase Reference

Initialization (Phases 1-5)

Content Processing (Phases 6-12)

Rendering (Phases 13-16)

Finalization (Phases 17-21)

Orchestrators

IncrementalOrchestrator Details

TheIncrementalOrchestratorwires the incremental pipeline into Phase 5:

The coordinator ensures that when a dependency changes (data file, template, taxonomy), all affected cache layers are invalidated consistently. See Cache for details.

Detection Pipeline (Phase 5)

Phase 5 (phase_incremental_filter) uses the provenance-based filter (bengal/orchestration/build/provenance_filter.py) and composes detectors frombengal/build/ (e.g., bengal/build/detectors/):

• DetectionPipeline composes ordered ChangeDetectorimplementations.
• Each detector returns aChangeDetectionResult, merged as the pipeline runs.
• Full rebuild triggers short-circuit the pipeline for clarity and speed.

This replaces the older monolithic filter engine and keeps detection logic testable, composable, and aligned with provenance-based builds.

BuildContext

Shared context passed through rendering and post-processing (simplified view):

@dataclass
class BuildContext:
    # Core (required by most phases)
    site: Site | None = None
    stats: BuildStats | None = None

    # Build mode flags
    incremental: bool = False
    verbose: bool = False
    quiet: bool = False
    strict: bool = False
    parallel: bool = True

    # Work items (populated during filtering)
    pages: list[Page] | None = None
    pages_to_build: list[Page] | None = None
    assets_to_process: list[Asset] | None = None
    changed_page_paths: set[Path] = field(default_factory=set)

    # Cache and tracking
    cache: BuildCache | None = None
    tracker: DependencyTracker | None = None

    # Output/progress
    cli: CLIOutput | None = None
    reporter: ProgressReporter | None = None
    output_collector: OutputCollector | None = None

Created early (during build setup) and enriched through the build. See bengal/orchestration/build_context.pyfor the complete definition.

Parallelization

Orchestrators auto-switch based on workload. Thresholds vary by subsystem:

# Incremental detectors (file/template change detection)
PARALLEL_THRESHOLD = 50

# Health check validators  
PARALLEL_THRESHOLD = 3

# General pattern
if parallel and len(items) > PARALLEL_THRESHOLD:
    with ThreadPoolExecutor() as executor:
        # Parallel processing
else:
    # Sequential for small workloads

The threshold varies because thread pool overhead differs by operation type—I/O-bound operations benefit from lower thresholds than CPU-bound ones.

Dashboard Notifications

The build broadcasts phase status for monitoring:

notify_phase_start("discovery")
# ... phase work ...
notify_phase_complete("discovery", duration_ms, "150 pages, 12 sections")

Groups: discovery, content, assets, rendering, finalization, health