Module

orchestration.incremental

Incremental build orchestration for Bengal SSG.

Handles cache management, change detection, and determining what needs rebuilding. Uses file hashes, dependency graphs, and taxonomy indexes to identify changed content and minimize rebuild work. Supports both full and incremental builds.

Key Concepts:

  • Change detection: File hash comparison for content changes
  • Dependency tracking: Template and data file dependencies
  • Taxonomy invalidation: Tag/category change detection
  • Selective rebuilding: Only rebuild changed pages and dependencies

Related Modules:

  • bengal.cache.build_cache: Build cache persistence
  • bengal.cache.dependency_tracker: Dependency graph construction
  • bengal.orchestration.build: Build orchestration entry point

See Also:

  • bengal/orchestration/incremental.py: IncrementalOrchestrator for incremental logic
  • plan/active/rfc-incremental-builds.md: Incremental build design
View source
1 Class

Classes

IncrementalOrchestrator
Orchestrates incremental build logic for efficient rebuilds. Handles cache initialization, change …
14

Orchestrates incremental build logic for efficient rebuilds.

Handles cache initialization, change detection, dependency tracking, and selective rebuilding. Uses file hashes, dependency graphs, and taxonomy indexes to minimize rebuild work by only rebuilding changed content.

Creation:

Direct instantiation: IncrementalOrchestrator(site)
  • Created by BuildOrchestrator when incremental builds enabled
  • Requires Site instance with content populated

Attributes

Name Type Description
site —

Site instance for incremental builds
cache —

BuildCache instance for build state persistence
tracker —

DependencyTracker instance for dependency graph construction
logger —

Logger instance for incremental build events
Relationships —
  • Uses: BuildCache for build state persistence - Uses: DependencyTracker for dependency graph construction - Used by: BuildOrchestrator for incremental build coordination - Uses: Site for content access and change detection Thread Safety: Not thread-safe. Should be used from single thread during build. Cache and tracker operations are thread-safe internally.

Methods 7

initialize
Initialize cache and dependency tracker for incremental builds. Sets up BuildC…
1 tuple[BuildCache, D… 
def initialize(self, enabled: bool = False) -> tuple[BuildCache, DependencyTracker]

Initialize cache and dependency tracker for incremental builds.

Sets up BuildCache and DependencyTracker instances. If enabled, loads existing cache from .bengal/cache.json (migrates from legacy location if needed). If disabled, creates empty cache instances.

Parameters 1
enabled bool

Whether incremental builds are enabled. If False, creates empty cache instances (full rebuilds always).
Returns

tuple[BuildCache, DependencyTracker] —

Tuple of (BuildCache, DependencyTracker) instances

Process:

1. Create .bengal/ directory if enabled
2. Migrate legacy cache from output_dir/.bengal-cache.json if exists
3. Load or create BuildCache instance
4. Create DependencyTracker with cache and site

check_config_changed
Check if configuration has changed (requires full rebuild). Uses config hash v…
0 bool 
def check_config_changed(self) -> bool

Check if configuration has changed (requires full rebuild).

Uses config hash validation which captures the effective configuration state:

  • Base config from files (bengal.toml, config/*.yaml)
  • Environment variable overrides (BENGAL_*)
  • Build profile settings (--profile writer)

This is more robust than file-based tracking because it detects changes in split config files, env vars, and profiles that wouldn't trigger a file hash change.

Returns

bool —

True if config changed (cache was invalidated)

find_work_early
Find pages/assets that need rebuilding (early version - before taxonomy generat…
3 tuple[list[Page], l… 
def find_work_early(self, verbose: bool = False, forced_changed_sources: set[Path] | None = None, nav_changed_sources: set[Path] | None = None) -> tuple[list[Page], list[Asset], ChangeSummary]

Find pages/assets that need rebuilding (early version - before taxonomy generation).

This is called BEFORE taxonomies/menus are generated, so it only checks content/asset changes. Generated pages (tags, etc.) will be determined later based on affected tags.

Uses section-level optimization: skips checking individual pages in unchanged sections.

Parameters 3
verbose bool

Whether to collect detailed change information
forced_changed_sources set[Path] | None
nav_changed_sources set[Path] | None
Returns

tuple[list[Page], list[Asset], ChangeSummary] —

Tuple of (pages_to_build, assets_to_process, change_summary) where change_summary is a ChangeSummary dataclass (supports dict-like access for compatibility)

find_work
Find pages/assets that need rebuilding (legacy version - after taxonomy generat…
1 tuple[list[Page], l… 
def find_work(self, verbose: bool = False) -> tuple[list[Page], list[Asset], dict[str, list[Any]]]

Find pages/assets that need rebuilding (legacy version - after taxonomy generation).

This is the old method that expects generated pages to already exist. Kept for backward compatibility but should be replaced with find_work_early().

Parameters 1
verbose bool

Whether to collect detailed change information
Returns

tuple[list[Page], list[Asset], dict[str, list[Any]]] —

Tuple of (pages_to_build, assets_to_process, change_summary)

process
Bridge-style process for testing incremental invalidation. ⚠️ TEST BRIDGE ONL…
2 None 
def process(self, change_type: str, changed_paths: set[str]) -> None

Bridge-style process for testing incremental invalidation.

⚠️ TEST BRIDGE ONLY

This method is a lightweight adapter used in tests to simulate an incremental pass without invoking the entire site build orchestrator.

Not for production use:

  • Writes placeholder output ("Updated") for verification only
  • Does not perform full rendering or asset processing
  • Skips postprocessing (RSS, sitemap, etc.)
  • Use run() or full_build() for production builds

Primarily consumed by:

  • tests/integration/test_full_to_incremental_sequence.py
  • bengal/orchestration/full_to_incremental.py (test bridge helper)
  • Test scenarios validating cache invalidation logic
Parameters 2
change_type str

One of "content", "template", or "config"
changed_paths set[str]

Set of paths that changed (ignored for "config")
full_rebuild
2 None 
def full_rebuild(self, pages: list[Any], context: BuildContext) -> None
Parameters 2
pages list[Any]
context BuildContext
save_cache
Update cache with processed files.
2 None 
def save_cache(self, pages_built: list[Page], assets_processed: list[Asset]) -> None

Update cache with processed files.

Parameters 2
pages_built list[Page]

Pages that were built
assets_processed list[Asset]

Assets that were processed
Internal Methods 7
__init__
Initialize incremental orchestrator.
1 None 
def __init__(self, site: Site)

Initialize incremental orchestrator.

Parameters 1
site Site

Site instance for incremental builds
_get_changed_sections
Identify sections with any changed files (section-level optimization). Uses ma…
1 set[Section] 
def _get_changed_sections(self, sections: list[Section] | None = None) -> set[Section]

Identify sections with any changed files (section-level optimization).

Uses max mtime of pages in each section to quickly skip entire sections that haven't changed. This is a major optimization for large sites where only a few sections have changes.

Parameters 1
sections list[Section] | None

List of sections to check. If None, uses site.sections.
Returns

set[Section] —

Set of Section objects that have changed files

Performance:

  • O(sections) instead of O(pages) for initial filtering
  • Only checks individual pages in changed sections
  • Uses fast mtime+size check from cache

_write_output
Write a placeholder output file corresponding to a content path. ⚠️ TEST HELP…
2 None 
def _write_output(self, path: Path, context: BuildContext) -> None

Write a placeholder output file corresponding to a content path.

⚠️ TEST HELPER - Used by process() bridge only.

For tests that exercise the bridge-only flow, derive the output location from the content path under the site's content dir. Writes diagnostic placeholder content for test verification.

Parameters 2
path Path

Source content path
context BuildContext

Build context (not used in this simplified version)
_cleanup_deleted_autodoc_sources
Clean up autodoc pages when their source files are deleted. Checks tracked aut…
0 None 
def _cleanup_deleted_autodoc_sources(self) -> None

Clean up autodoc pages when their source files are deleted.

Checks tracked autodoc source files and removes corresponding output when the source no longer exists. This prevents stale autodoc pages from remaining when Python/OpenAPI source files are deleted.

_cleanup_deleted_files
Clean up output files for deleted source files. Checks cache for source files …
0 None 
def _cleanup_deleted_files(self) -> None

Clean up output files for deleted source files.

Checks cache for source files that no longer exist and deletes their corresponding output files. This prevents stale content from remaining in the output directory after source deletion.

_find_cascade_affected_pages
Find all pages affected by a cascade change in a section index. When a section…
1 set[Path] 
def _find_cascade_affected_pages(self, index_page: Page) -> set[Path]

Find all pages affected by a cascade change in a section index.

When a section's _index.md has cascade metadata and is modified, all descendant pages inherit those values and need to be rebuilt.

Parameters 1
index_page Page

Section _index.md page with cascade metadata
Returns

set[Path] —

Set of page source paths that should be rebuilt due to cascade

_get_theme_templates_dir
Get the templates directory for the current theme.
0 Path | None 
def _get_theme_templates_dir(self) -> Path | None

Get the templates directory for the current theme.

Returns

Path | None —

Path to theme templates or None if not found