# core

URL: /api/rendering/pipeline/core/
Section: pipeline

--------------------------------------------------------------------------------

core - Bengal window.BENGAL_THEME_DEFAULTS = { appearance: 'dark', palette: 'snow-lynx' }; // Progressive Enhancement System Configuration window.Bengal = window.Bengal || {}; window.Bengal.enhanceBaseUrl = '/bengal/assets/js/enhancements'; window.Bengal.watchDom = true; window.Bengal.debug = false; (function () { try { var defaults = window.BENGAL_THEME_DEFAULTS || { appearance: 'system', palette: '' }; var defaultAppearance = defaults.appearance; if (defaultAppearance === 'system') { defaultAppearance = (window.matchMedia && window.matchMedia('(prefers-color-scheme: dark)').matches) ? 'dark' : 'light'; } var storedTheme = localStorage.getItem('bengal-theme'); var storedPalette = localStorage.getItem('bengal-palette'); var theme = storedTheme ? (storedTheme === 'system' ? defaultAppearance : storedTheme) : defaultAppearance; var palette = storedPalette ?? defaults.palette; document.documentElement.setAttribute('data-theme', theme); if (palette) { document.documentElement.setAttribute('data-palette', palette); } } catch (e) { document.documentElement.setAttribute('data-theme', 'light'); } })(); Skip to main content Magnifying Glass ESC Recent Clear Magnifying Glass No results for "" Try different keywords or check your spelling Start typing to search... ↑↓ Navigate ↵ Open ESC Close Powered by Lunr ᓚᘏᗢ Documentation Info About Arrow Clockwise Get Started Note Tutorials File Text Content Palette Theming Settings Building Starburst Extending Bookmark Reference Learning Tracks Releases Dev GitHub API Reference bengal CLI Magnifying Glass Search ⌘K Palette Appearance Chevron Down Mode Monitor System Sun Light Moon Dark Palette Snow Lynx Brown Bengal Silver Bengal Charcoal Bengal Blue Bengal List ᓚᘏᗢ Magnifying Glass Search X Close Documentation Info About Arrow Clockwise Get Started Note Tutorials File Text Content Palette Theming Settings Building Starburst Extending Bookmark Reference Learning Tracks Releases Dev GitHub API Reference bengal CLI Palette Appearance Chevron Down Mode Monitor System Sun Light Moon Dark Palette Snow Lynx Brown Bengal Silver Bengal Charcoal Bengal Blue Bengal API Reference __main__ bengal Caret Right Folder Analysis community_detection graph_analysis graph_reporting graph_visualizer knowledge_graph link_suggestions link_types page_rank path_analysis performance_advisor results Caret Right Folder Assets manifest pipeline Caret Right Folder Autodoc base config docstring_parser utils virtual_orchestrator Caret Right Folder Extractors cli openapi python Caret Right Folder Models cli common openapi python Caret Right Folder Cache asset_dependency_map cache_store cacheable compression dependency_tracker page_discovery_cache query_index query_index_registry taxonomy_index utils Caret Right Folder Build Cache autodoc_tracking core file_tracking fingerprint parsed_content_cache rendered_output_cache taxonomy_index_mixin validation_cache Caret Right Folder Indexes author_index category_index date_range_index section_index Caret Right Folder Cli __main__ base site_templates utils Caret Right Folder Commands assets build clean collections config debug explain fix health init perf project serve site skeleton sources theme utils validate Caret Right Folder Graph __main__ bridges communities orphans pagerank report suggest Caret Right Folder New config presets scaffolds site wizard Caret Right Folder Helpers cli_app_loader cli_output config_validation error_handling menu_config metadata progress site_loader traceback validation Caret Right Folder Skeleton hydrator schema Caret Right Folder Templates base registry Caret Right Folder Blog template Caret Right Folder Changelog template Caret Right Folder Default template Caret Right Folder Docs template Caret Right Folder Landing template Caret Right Folder Portfolio template Caret Right Folder Resume template Caret Right Folder Collections errors loader schemas validator Caret Right Folder Config defaults deprecation directory_loader env_overrides environment feature_mappings hash loader merge origin_tracker validators Caret Right Folder Content Layer entry loaders manager source Caret Right Folder Sources github local notion rest Caret Right Folder Content Types base registry strategies Caret Right Folder Core build_context cascade_engine menu section theme Caret Right Folder Asset asset_core css_transforms Caret Right Folder Page computed content metadata navigation operations page_core proxy relationships utils Caret Right Folder Site core data discovery factories page_caches properties section_registry theme Caret Right Folder Debug base config_inspector content_migrator delta_analyzer dependency_visualizer explainer incremental_debugger models reporter shortcode_sandbox Caret Right Folder Discovery asset_discovery content_discovery Caret Right Folder Fonts downloader generator Caret Right Folder Health autofix base health_check report Caret Right Folder Linkcheck async_checker ignore_policy internal_checker models orchestrator Caret Right Folder Validators anchors assets cache config connectivity cross_ref fonts links menu navigation output performance rendering rss sitemap taxonomy tracks Caret Right Folder Directives analysis checkers constants Caret Right Folder Orchestration asset content full_to_incremental incremental menu postprocess related_posts render section static streaming taxonomy Caret Right Folder Postprocess html_output redirects rss sitemap special_pages Caret Right Folder Output Formats index_generator json_generator llm_generator lunr_index_generator txt_generator utils Caret Right Folder Rendering api_doc_enhancer asset_extractor errors jinja_utils link_transformer link_validator pygments_cache renderer template_context template_profiler validator Caret Right Folder Parsers base factory mistune native_html pygments_patch python_markdown Caret Right Folder Pipeline core output thread_local toc transforms Caret Right Folder Plugins badges cross_references inline_icon term variable_substitution Caret Right Folder Directives _icons admonitions badge base button cache cards checklist code_tabs container contracts data_table dropdown embed errors example_label fenced figure glossary icon include list_table literalinclude marimo navigation options rubric steps tabs target term terminal tokens utils validator video Caret Right Folder Template Engine asset_url core environment manifest menu url_helpers Caret Right Folder Template Functions advanced_collections advanced_strings autodoc collections content crossref data dates debug files get_page i18n icons images math_functions navigation pagination_helpers seo strings tables taxonomies theme urls Caret Right Folder Server build_handler component_preview constants dev_server live_reload pid_manager reload_controller request_handler request_logger resource_manager utils Caret Right Folder Services validation Caret Right Folder Themes config Caret Right Folder Utils atomic_write autodoc build_context build_stats build_summary cli_output css_minifier dates dotdict error_handlers file_io file_lock hashing incremental_constants js_bundler live_progress logger metadata observability page_initializer pagination path_resolver paths performance_collector performance_report profile progress retry rich_console sections swizzle text theme_registry theme_resolution thread_local traceback_config traceback_renderer url_normalization url_strategy Rendering Pipeline ᗢ Caret Down Link Copy URL External Open LLM text Copy Copy LLM text Share with AI Ask Claude Ask ChatGPT Ask Gemini Ask Copilot Module rendering.pipeline.core Core rendering pipeline for Bengal SSG. Orchestrates the parsing, AST building, templating, and output rendering phases for individual pages. Manages thread-local parser instances for performance and provides dependency tracking for incremental builds. Key Concepts: Thread-local parsers: Parser instances reused per thread for performance AST-based processing: Content represented as AST for efficient transformation Template rendering: Jinja2 template rendering with page context Dependency tracking: Template and asset dependency tracking Related Modules: bengal.rendering.parsers.mistune: Markdown parser implementation bengal.rendering.template_engine: Template engine for Jinja2 rendering bengal.rendering.renderer: Individual page rendering logic bengal.cache.dependency_tracker: Dependency graph construction See Also: plan/active/rfc-content-ast-architecture.md: AST architecture RFC View source 2 Classes 1 Function Classes RenderingPipeline Coordinates the entire rendering process for content pages. Orchestrates the complete rendering pi… 21 Caret Right Coordinates the entire rendering process for content pages. Orchestrates the complete rendering pipeline from markdown parsing through template rendering to final HTML output. Manages thread-local parser instances for performance and integrates with dependency tracking for incremental builds. Creation: Direct instantiation: RenderingPipeline(site, dependency_tracker=None, ...) Created by RenderOrchestrator for page rendering One instance per worker thread (thread-local) Requires Site instance with config Attributes Name Type Description site — Site instance with config and xref_index parser — Thread-local markdown parser (cached per thread) dependency_tracker — Optional DependencyTracker for incremental builds quiet — Whether to suppress per-page output build_stats — Optional BuildStats for error collection Pipeline Stages: 1. Parse source content (Markdown, etc.) 2. Build Abstract Syntax Tree (AST) 3. Apply templates (Jinja2) 4. Render output (HTML) 5. Write to output directory Relationships — Uses: TemplateEngine for template rendering - Uses: Renderer for individual page rendering - Uses: DependencyTracker for dependency tracking - Used by: RenderOrchestrator for page rendering Thread Safety: Thread-safe. Uses thread-local parser instances. Each thread should have its own RenderingPipeline instance. Methods 1 process_page Process a single page through the entire rendering pipeline. Executes all rend… 1 None Caret Right def process_page(self, page: Page) -> None Process a single page through the entire rendering pipeline. Executes all rendering stages: parsing, AST building, template rendering, and output writing. Uses cached parsed content when available. Virtual pages (e.g., autodoc API pages) bypass markdown parsing and use pre-rendered HTML directly. Parameters 1 page Page Page object to process. Must have source_path set. Internal Methods 20 Caret Right __init__ Initialize the rendering pipeline. Parser Selection: Reads from config in … 6 None Caret Right def __init__(self, site: Any, dependency_tracker: Any = None, quiet: bool = False, build_stats: Any = None, build_context: Any | None = None, changed_sources: set[Path] | None = None) -> None Initialize the rendering pipeline. Parser Selection: Reads from config in this order: 1. config['markdown_engine'] (legacy) 2. config['markdown']['parser'] (preferred) 3. Default: 'mistune' (recommended for speed) Parser Caching: Uses thread-local caching via get_thread_parser(). Creates ONE parser per worker thread, cached for reuse. Parameters 6 site Any Site instance with config and xref_index dependency_tracker Any Optional tracker for incremental builds quiet bool If True, suppress per-page output build_stats Any Optional BuildStats object to collect warnings build_context Any | None Optional BuildContext for dependency injection changed_sources set[Path] | None _try_rendered_cache Try to use rendered output cache. Returns True if cache hit. 2 bool Caret Right def _try_rendered_cache(self, page: Page, template: str) -> bool Try to use rendered output cache. Returns True if cache hit. Parameters 2 page Page template str Returns bool _try_parsed_cache Try to use parsed content cache. Returns True if cache hit. 3 bool Caret Right def _try_parsed_cache(self, page: Page, template: str, parser_version: str) -> bool Try to use parsed content cache. Returns True if cache hit. Parameters 3 page Page template str parser_version str Returns bool _parse_content Parse page content through markdown parser. 1 None Caret Right def _parse_content(self, page: Page) -> None Parse page content through markdown parser. Parameters 1 page Page _should_generate_toc Determine if TOC should be generated for this page. 1 bool Caret Right def _should_generate_toc(self, page: Page) -> bool Determine if TOC should be generated for this page. Parameters 1 page Page Returns bool _parse_with_mistune Parse content using Mistune parser. 2 None Caret Right def _parse_with_mistune(self, page: Page, need_toc: bool) -> None Parse content using Mistune parser. Parameters 2 page Page need_toc bool _parse_with_legacy Parse content using legacy python-markdown parser. 2 None Caret Right def _parse_with_legacy(self, page: Page, need_toc: bool) -> None Parse content using legacy python-markdown parser. Parameters 2 page Page need_toc bool _enhance_api_docs Enhance API documentation with badges. 1 None Caret Right def _enhance_api_docs(self, page: Page) -> None Enhance API documentation with badges. Parameters 1 page Page _cache_parsed_content Store parsed content in cache for next build. 3 None Caret Right def _cache_parsed_content(self, page: Page, template: str, parser_version: str) -> None Store parsed content in cache for next build. Parameters 3 page Page template str parser_version str _render_and_write Render template and write output. 2 None Caret Right def _render_and_write(self, page: Page, template: str) -> None Render template and write output. Parameters 2 page Page template str _cache_rendered_output Store rendered output in cache for next build. 2 None Caret Right def _cache_rendered_output(self, page: Page, template: str) -> None Store rendered output in cache for next build. Parameters 2 page Page template str _accumulate_json_data Accumulate JSON data during rendering for post-processing optimization. 1 None Caret Right def _accumulate_json_data(self, page: Page) -> None Accumulate JSON data during rendering for post-processing optimization. Parameters 1 page Page _process_virtual_page Process a virtual page with pre-rendered HTML content. Virtual pages (like aut… 1 None Caret Right def _process_virtual_page(self, page: Page) -> None Process a virtual page with pre-rendered HTML content. Virtual pages (like autodoc pages) may have pre-rendered HTML that is either: A complete HTML page (extends base.html) - use directly A content fragment - wrap with template Deferred autodoc page - render now with full context (menus available) Complete pages start with <!DOCTYPE or <html and should not be wrapped. Parameters 1 page Page _render_autodoc_page Render an autodoc page using the site's template engine. This is called during… 1 None Caret Right def _render_autodoc_page(self, page: Page) -> None Render an autodoc page using the site's template engine. This is called during the rendering phase (after menus are built), ensuring full template context is available for proper header/nav rendering. Parameters 1 page Page Virtual page with autodoc_element in metadata _build_variable_context Build variable context for {{ variable }} substitution in markdown. 1 dict[str, Any] Caret Right def _build_variable_context(self, page: Page) -> dict[str, Any] Build variable context for {{ variable }} substitution in markdown. Parameters 1 page Page Returns dict[str, Any] _get_parser_version Get parser version string for cache validation. 0 str Caret Right def _get_parser_version(self) -> str Get parser version string for cache validation. Returns str _write_output Write rendered page to output directory (backward compatibility wrapper). 1 None Caret Right def _write_output(self, page: Page) -> None Write rendered page to output directory (backward compatibility wrapper). Parameters 1 page Page _preprocess_content Pre-process page content through Jinja2 (legacy parser only). 1 str Caret Right def _preprocess_content(self, page: Page) -> str Pre-process page content through Jinja2 (legacy parser only). Parameters 1 page Page Returns str _normalize_autodoc_element Ensure autodoc element metadata supports both dotted and mapping access. 1 Any Caret Right def _normalize_autodoc_element(self, element: Any) -> Any Ensure autodoc element metadata supports both dotted and mapping access. Parameters 1 element Any Returns Any _normalize_config Wrap config to allow dotted access with safe defaults for github metadata. Ext… 1 Any Caret Right def _normalize_config(self, config: Any) -> Any Wrap config to allow dotted access with safe defaults for github metadata. Extracts github_repo and github_branch from the autodoc section of the config and normalizes github_repo to a full URL if provided in owner/repo format. Parameters 1 config Any Returns Any _MetadataView Dict that also supports attribute-style access (dotted) used by templates. 1 Caret Right Dict that also supports attribute-style access (dotted) used by templates. Inherits from dict[str, Any] Internal Methods 1 Caret Right __getattr__ 1 Any Caret Right def __getattr__(self, item: str) -> Any Parameters 1 item str Returns Any Functions _safe_metadata_summary Summarize metadata for logging without raising on missing attributes. 1 str Caret Right def _safe_metadata_summary(meta: Any) -> str Summarize metadata for logging without raising on missing attributes. Parameters 1 Name Type Default Description meta Any — Returns str Next → output List © 2025 Bengal ᓚᘏᗢ window.BENGAL_LAZY_ASSETS = { tabulator: '/bengal/assets/js/tabulator.min.js', dataTable: '/bengal/assets/js/data-table.js', mermaidToolbar: '/bengal/assets/js/mermaid-toolbar.9de5abba.js', mermaidTheme: '/bengal/assets/js/mermaid-theme.344822c5.js', graphMinimap: '/bengal/assets/js/graph-minimap.cc7e42e3.js', graphContextual: '/bengal/assets/js/graph-contextual.440e59c6.js' }; window.BENGAL_ICONS = { close: '/bengal/assets/icons/close.911d4fe1.svg', enlarge: '/bengal/assets/icons/enlarge.652035e5.svg', copy: '/bengal/assets/icons/copy.3d56e945.svg', 'download-svg': '/bengal/assets/icons/download.04f07e1b.svg', 'download-png': '/bengal/assets/icons/image.c34dfd40.svg', 'zoom-in': '/bengal/assets/icons/zoom-in.237b4a83.svg', 'zoom-out': '/bengal/assets/icons/zoom-out.38857c77.svg', reset: '/bengal/assets/icons/reset.d26dba29.svg' }; Arrow Up X

--------------------------------------------------------------------------------

Metadata:
- Author: lbliii
- Word Count: 2262
- Reading Time: 11 minutes