Module

rendering.pipeline.core

Core rendering pipeline for Bengal SSG.

Orchestrates the parsing, AST building, templating, and output rendering phases for individual pages. Manages thread-local parser instances for performance and provides dependency tracking for incremental builds.

Key Concepts:

  • Thread-local parsers: Parser instances reused per thread for performance
  • AST-based processing: Content represented as AST for efficient transformation
  • Template rendering: Jinja2 template rendering with page context
  • Dependency tracking: Template and asset dependency tracking

Related Modules:

  • bengal.rendering.parsers.mistune: Markdown parser implementation
  • bengal.rendering.template_engine: Template engine for Jinja2 rendering
  • bengal.rendering.renderer: Individual page rendering logic
  • bengal.cache.dependency_tracker: Dependency graph construction

See Also:

  • plan/active/rfc-content-ast-architecture.md: AST architecture RFC
View source
2 Classes 1 Function

Classes

RenderingPipeline
Coordinates the entire rendering process for content pages. Orchestrates the complete rendering pi…
21

Coordinates the entire rendering process for content pages.

Orchestrates the complete rendering pipeline from markdown parsing through template rendering to final HTML output. Manages thread-local parser instances for performance and integrates with dependency tracking for incremental builds.

Creation:

Direct instantiation: RenderingPipeline(site, dependency_tracker=None, ...)
  • Created by RenderOrchestrator for page rendering
  • One instance per worker thread (thread-local)
  • Requires Site instance with config

Attributes

Name Type Description
site —

Site instance with config and xref_index
parser —

Thread-local markdown parser (cached per thread)
dependency_tracker —

Optional DependencyTracker for incremental builds
quiet —

Whether to suppress per-page output
build_stats —

Optional BuildStats for error collection Pipeline Stages: 1. Parse source content (Markdown, etc.) 2. Build Abstract Syntax Tree (AST) 3. Apply templates (Jinja2) 4. Render output (HTML) 5. Write to output directory
Relationships —
  • Uses: TemplateEngine for template rendering - Uses: Renderer for individual page rendering - Uses: DependencyTracker for dependency tracking - Used by: RenderOrchestrator for page rendering Thread Safety: Thread-safe. Uses thread-local parser instances. Each thread should have its own RenderingPipeline instance.

Methods 1

process_page
Process a single page through the entire rendering pipeline. Executes all rend…
1 None 
def process_page(self, page: Page) -> None

Process a single page through the entire rendering pipeline.

Executes all rendering stages: parsing, AST building, template rendering, and output writing. Uses cached parsed content when available.

Virtual pages (e.g., autodoc API pages) bypass markdown parsing and use pre-rendered HTML directly.

Parameters 1
page Page

Page object to process. Must have source_path set.
Internal Methods 20
__init__
Initialize the rendering pipeline. Parser Selection: Reads from config in …
6 None 
def __init__(self, site: Any, dependency_tracker: Any = None, quiet: bool = False, build_stats: Any = None, build_context: Any | None = None, changed_sources: set[Path] | None = None) -> None

Initialize the rendering pipeline.

Parser Selection:

Reads from config in this order:
1. config['markdown_engine'] (legacy)
2. config['markdown']['parser'] (preferred)
3. Default: 'mistune' (recommended for speed)

Parser Caching:

Uses thread-local caching via get_thread_parser().
Creates ONE parser per worker thread, cached for reuse.
Parameters 6
site Any

Site instance with config and xref_index
dependency_tracker Any

Optional tracker for incremental builds
quiet bool

If True, suppress per-page output
build_stats Any

Optional BuildStats object to collect warnings
build_context Any | None

Optional BuildContext for dependency injection
changed_sources set[Path] | None
_try_rendered_cache
Try to use rendered output cache. Returns True if cache hit.
2 bool 
def _try_rendered_cache(self, page: Page, template: str) -> bool

Try to use rendered output cache. Returns True if cache hit.

Parameters 2
page Page
template str
Returns

bool

_try_parsed_cache
Try to use parsed content cache. Returns True if cache hit.
3 bool 
def _try_parsed_cache(self, page: Page, template: str, parser_version: str) -> bool

Try to use parsed content cache. Returns True if cache hit.

Parameters 3
page Page
template str
parser_version str
Returns

bool

_parse_content
Parse page content through markdown parser.
1 None 
def _parse_content(self, page: Page) -> None

Parse page content through markdown parser.

Parameters 1
page Page
_should_generate_toc
Determine if TOC should be generated for this page.
1 bool 
def _should_generate_toc(self, page: Page) -> bool

Determine if TOC should be generated for this page.

Parameters 1
page Page
Returns

bool

_parse_with_mistune
Parse content using Mistune parser.
2 None 
def _parse_with_mistune(self, page: Page, need_toc: bool) -> None

Parse content using Mistune parser.

Parameters 2
page Page
need_toc bool
_parse_with_legacy
Parse content using legacy python-markdown parser.
2 None 
def _parse_with_legacy(self, page: Page, need_toc: bool) -> None

Parse content using legacy python-markdown parser.

Parameters 2
page Page
need_toc bool
_enhance_api_docs
Enhance API documentation with badges.
1 None 
def _enhance_api_docs(self, page: Page) -> None

Enhance API documentation with badges.

Parameters 1
page Page
_cache_parsed_content
Store parsed content in cache for next build.
3 None 
def _cache_parsed_content(self, page: Page, template: str, parser_version: str) -> None

Store parsed content in cache for next build.

Parameters 3
page Page
template str
parser_version str
_render_and_write
Render template and write output.
2 None 
def _render_and_write(self, page: Page, template: str) -> None

Render template and write output.

Parameters 2
page Page
template str
_cache_rendered_output
Store rendered output in cache for next build.
2 None 
def _cache_rendered_output(self, page: Page, template: str) -> None

Store rendered output in cache for next build.

Parameters 2
page Page
template str
_accumulate_json_data
Accumulate JSON data during rendering for post-processing optimization.
1 None 
def _accumulate_json_data(self, page: Page) -> None

Accumulate JSON data during rendering for post-processing optimization.

Parameters 1
page Page
_process_virtual_page
Process a virtual page with pre-rendered HTML content. Virtual pages (like aut…
1 None 
def _process_virtual_page(self, page: Page) -> None

Process a virtual page with pre-rendered HTML content.

Virtual pages (like autodoc pages) may have pre-rendered HTML that is either:

  1. A complete HTML page (extends base.html) - use directly
  2. A content fragment - wrap with template
  3. Deferred autodoc page - render now with full context (menus available)

Complete pages start with <!DOCTYPE or <html and should not be wrapped.

Parameters 1
page Page
_render_autodoc_page
Render an autodoc page using the site's template engine. This is called during…
1 None 
def _render_autodoc_page(self, page: Page) -> None

Render an autodoc page using the site's template engine.

This is called during the rendering phase (after menus are built), ensuring full template context is available for proper header/nav rendering.

Parameters 1
page Page

Virtual page with autodoc_element in metadata
_build_variable_context
Build variable context for {{ variable }} substitution in markdown.
1 dict[str, Any] 
def _build_variable_context(self, page: Page) -> dict[str, Any]

Build variable context for {{ variable }} substitution in markdown.

Parameters 1
page Page
Returns

dict[str, Any]

_get_parser_version
Get parser version string for cache validation.
0 str 
def _get_parser_version(self) -> str

Get parser version string for cache validation.

Returns

str

_write_output
Write rendered page to output directory (backward compatibility wrapper).
1 None 
def _write_output(self, page: Page) -> None

Write rendered page to output directory (backward compatibility wrapper).

Parameters 1
page Page
_preprocess_content
Pre-process page content through Jinja2 (legacy parser only).
1 str 
def _preprocess_content(self, page: Page) -> str

Pre-process page content through Jinja2 (legacy parser only).

Parameters 1
page Page
Returns

str

_normalize_autodoc_element
Ensure autodoc element metadata supports both dotted and mapping access.
1 Any 
def _normalize_autodoc_element(self, element: Any) -> Any

Ensure autodoc element metadata supports both dotted and mapping access.

Parameters 1
element Any
Returns

Any

_normalize_config
Wrap config to allow dotted access with safe defaults for github metadata. Ext…
1 Any 
def _normalize_config(self, config: Any) -> Any

Wrap config to allow dotted access with safe defaults for github metadata.

Extracts github_repo and github_branch from the autodoc section of the config and normalizes github_repo to a full URL if provided in owner/repo format.

Parameters 1
config Any
Returns

Any

_MetadataView
Dict that also supports attribute-style access (dotted) used by templates.
1

Dict that also supports attribute-style access (dotted) used by templates.

Inherits from dict[str, Any]
Internal Methods 1
__getattr__
1 Any 
def __getattr__(self, item: str) -> Any
Parameters 1
item str
Returns

Any

Functions

_safe_metadata_summary
Summarize metadata for logging without raising on missing attributes.
1 str 
def _safe_metadata_summary(meta: Any) -> str

Summarize metadata for logging without raising on missing attributes.

Parameters 1

Name Type Default Description
meta Any —

Returns

str