Performance Optimizer

Optimize Build Performance

Speed up Bengal with incremental builds, parallel processing, and smart caching.

Do I Need This?

Quick Wins

Bengal optimizes builds automatically — no configuration required:

• Parallel processing: Auto-enabled based on page count and CPU cores
• Incremental builds: Auto-enabled when build cache exists
• Smart caching: Tracks file changes, dependencies, and parsed content
• Fast mode: Skips HTML formatting for faster builds (--fast or build.fast_mode)
• Render-time asset tracking: Tracks assets during template rendering (no HTML parsing)
• Autodoc AST caching: Caches parsed Python modules to skip AST parsing on subsequent builds

# bengal.toml — only needed to override defaults
[build]
max_workers = 8      # Limit parallel workers (default: auto-detect)

How Builds Get Faster

Performance Strategies

Common Commands

# Fast mode: quiet output, auto-parallel
bengal build --fast

# Force full rebuild (skip cache)
bengal build --no-incremental

# Clear build cache
bengal clean --cache

# Profile build time (saves to .bengal/profiles/)
bengal build --perf-profile

# Profile template rendering
bengal build --profile-templates

Large Site Optimization

Bengal is designed for sites with thousands of pages. This guide covers strategies for sites beyond 5,000 pages.

Quick Start

For sites with 5K+ pages:

# Memory-optimized build
bengal build --memory-optimized --fast

# Full incremental + parallel + fast
bengal build --incremental --fast

Strategy Overview

1. Memory-Optimized Builds (Streaming Mode)

For sites with 5K+ pages, enable streaming mode:

bengal build --memory-optimized

How It Works

1. Builds knowledge graph to understand page connectivity
2. Renders hubs first (highly connected pages) and keeps them in memory
3. Streams leaves in batches and releases memory immediately
4. Result: 80-90% memory reduction

When to Use

• Sites with 5K+ pages
• CI runners with limited memory
• Docker containers with memory limits
• Local machines with limited RAM

2. Query Indexes (O(1) Lookups)

Replace O(n) page filtering with O(1) index lookups in templates.

The Problem

{# O(n) - scans ALL pages on every request #}
{% let blog_posts = site.pages | where('section', 'blog') %}

On a 10K page site, this filter runs 10,000 comparisons.

The Solution

{# O(1) - instant hash lookup #}
{% let blog_posts = site.indexes.section.get('blog') | resolve_pages %}

Built-in Indexes

Usage Examples

Section-based listing:

{% let blog_posts = site.indexes.section.get('blog') | resolve_pages %}
{% for post in blog_posts | sort_by('date', reverse=true) %}
  <h2>{{ post.title }}</h2>
{% end %}

Author archive:

{% let author_posts = site.indexes.author.get('Jane Smith') | resolve_pages %}
<p>{{ author_posts | length }} posts by Jane</p>

Monthly archives:

{% let jan_posts = site.indexes.date_range.get('2024-01') | resolve_pages %}
{% for post in jan_posts %}
  {{ post.title }}
{% end %}

Performance Impact

3. Parallel Processing

Parallel processing is auto-detected based on page count and workload. Adjust worker count if needed:

# bengal.toml
[build]
max_workers = 8           # Optional: adjust based on CPU cores (auto-detected if omitted)

To force sequential processing (useful for debugging):

bengal build --no-parallel

Free-Threaded Python

Bengal automatically detects Python 3.14t+ (free-threaded):

# 1.5-2x faster rendering
# Install free-threaded Python:
pyenv install 3.14t
python3.14t -m pip install bengal

When running on free-threaded Python:

• ThreadPoolExecutor gets true parallelism (no GIL contention)
• ~1.78x faster rendering on multi-core machines
• No code changes needed

4. Incremental Builds

Incremental builds are automatic — no configuration needed. First build is full, subsequent builds only rebuild changed content. Force a full rebuild if needed:

# Force full rebuild (skip cache)
bengal build --no-incremental

What Gets Cached

• Content parsing — Markdown AST cached per file
• Template rendering — Output cached by content hash
• Asset hashing — Fingerprints cached
• Query indexes — Updated incrementally
• Autodoc AST parsing — Python modules cached to skip AST parsing (30-40% speedup for autodoc-heavy sites)
• Asset dependencies — Tracked during render-time (no HTML parsing needed)

Cache Location

.bengal/
├── cache.json.zst          # Main build cache (compressed)
├── page_metadata.json.zst  # Page discovery cache
├── taxonomy_index.json.zst # Taxonomy index
├── indexes/                # Query indexes (section, author, etc.)
├── templates/              # Template bytecode cache
└── logs/                   # Build logs

Clear Cache

# Clear all caches (forces cold rebuild)
bengal clean --cache

# Clear output and cache
bengal clean --all

5. Fast Mode

Combine all optimizations for maximum speed:

bengal build --fast

--fastenables:

• Quiet output (minimal console I/O)
• Suppresses verbose logging
• Parallelism auto-detected as normal
• Skips HTML formatting (raw HTML output, ~10-15% faster)

6. Build Profiling

Identify bottlenecks:

# Generate performance profile
bengal build --perf-profile

# View results
python -m pstats .bengal/profiles/profile.stats

Template Profiling

Find slow templates:

bengal build --profile-templates

Output:

Template Rendering Times:
  layouts/blog.html: 1.2s (340 pages, 3.5ms avg)
  layouts/docs.html: 0.8s (890 pages, 0.9ms avg)
  partials/nav.html: 0.3s (included 1230 times)

7. Content Organization

Split Large Sections

If one section has 5K+ pages, consider splitting:

content/
├── blog/
│   ├── 2024/     # 500 pages
│   ├── 2023/     # 800 pages
│   └── archive/  # 3000+ pages (separate pagination)

Use Pagination

Don't render 1000 items on one page:

# Paginate blog listing
pagination:
  per_page: 20

Lazy-Load Heavy Content

Move rarely-accessed content to separate pages:

{# Don't: render full changelog inline #}
{{ include('changelog.html') }}

{# Do: link to separate page #}
<a href="/changelog/">View full changelog</a>

8. CI/CD Optimization

GitHub Actions Example

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache Bengal
        uses: actions/cache@v4
        with:
          path: .bengal
          key: bengal-${{ hashFiles('content/**/*.md') }}

      - name: Build
        run: bengal build --fast --environment production

Docker Memory Limits

# Use memory-optimized for container builds
CMD ["bengal", "build", "--memory-optimized", "--fast"]

9. Monitoring Build Health

Track build performance over time:

# Detailed build stats
bengal build --verbose

Output:

Build Summary:
  Total Pages: 15,432
  Rendered: 342 (incremental)
  Skipped: 15,090 (cached)
  Duration: 12.3s
  Memory Peak: 245MB
  Pages/sec: 1,254

Quick Reference

# Memory-efficient large site build
bengal build --memory-optimized --fast

# Profile to find bottlenecks
bengal build --perf-profile --profile-templates

# Force full rebuild
bengal build --no-incremental

# Clear all caches
bengal clean --cache

# Clear output and cache
bengal clean --all

Troubleshooting

Build runs out of memory

1. Enable streaming:--memory-optimized
2. Usebengal build --dev --verboseto see memory usage
3. Increase swap space

Build is slow despite caching

1. Check what's invalidating cache:bengal build --verbose
2. Profile templates:--profile-templates
3. Check for O(n) filters in templates (use query indexes)

Incremental not working

1. Ensure.bengal/is not gitignored for local dev
2. Runbengal clean --cacheto reset
3. Check for template changes that invalidate all pages

Site Analysis

Analyze your site's structure to improve navigation and discoverability.

Do I Need This?

Analysis Tools

Quick Start

# Unified connectivity report
bengal graph report

# Brief output for CI
bengal graph report --brief

# CI mode with thresholds
bengal graph report --ci --threshold-isolated 5

# Find isolated pages
bengal graph orphans

# Find lightly-linked pages
bengal graph orphans --level lightly

# JSON output for processing
bengal graph orphans --format json

# Paths only (for scripting)
bengal graph orphans --format paths

bengal graph pagerank --top-n 20

Use Cases

Graph Analysis

Bengal's graph analysis tools help you understand your site's structure, identify optimization opportunities, and improve internal linking.

Overview

Bengal uses a semantic link model that understands different types of connections:

This weighted approach provides nuanced connectivity analysis beyond simple orphan detection, eliminating false positives from structural links.

Connectivity Levels

Pages are classified by their weighted connectivity score:

Quick Start

Get a unified analysis report:

bengal graph report

Example: Output

================================================================================
📊 Site Analysis Report
================================================================================

📈 Overview
   Total pages:        124
   Total links:        316
   Avg links/page:     2.5
   Avg conn. score:    1.46
   Communities:        0

🔗 Connectivity Distribution
   🟢 Well-Connected:      39 pages (31.5%)
   🟡 Adequately:          38 pages (30.6%)
   🟠 Lightly Linked:      26 pages (21.0%)
   🔴 Isolated:            21 pages (16.9%) ⚠️

🔴 Isolated Pages (need attention)
      • content/_index.md
      • content/docs/_index.md
      ... and 19 more

💡 Recommendations
   • Add explicit cross-references to isolated pages
   • Add internal links to lightly-linked pages
================================================================================

For CI pipelines:

bengal graph report --brief --ci --threshold-isolated 5

Example: CI Output

📊 Site Analysis: 124 pages
   Isolated: 21 (16.9%) ⚠️
   Lightly linked: 26 (21.0%)
   Avg score: 1.46 (good)
✅ CI PASSED: 21 isolated pages within threshold (25)

Understanding the Analysis

Semantic Link Types

Bengal tracks different types of links with semantic meaning:

Key Metrics

• Connectivity score: Weighted sum of all incoming links
• Average score: Mean connectivity score across all pages (aim for ≥1.0)
• Isolated pages: Pages with score < 0.25 (need attention)
• Distribution: Percentage of pages at each connectivity level
• Hub pages: Pages with many incoming links (important for navigation)

Command Reference

Unified Analysis Report

Get a comprehensive connectivity report:

bengal graph report

Options:

• --brief: Compact output for CI pipelines
• --ci: CI mode with exit codes
• --threshold-isolated N: Max isolated pages before failure (default: 5)
• --threshold-lightly N: Max lightly-linked before warning (default: 20)
• --format FORMAT: Output format - console, json, or markdown

List Under-Linked Pages

Find pages by connectivity level:

# Show isolated pages (default)
bengal graph orphans

# Show lightly-linked pages
bengal graph orphans --level lightly

# Show both isolated and lightly-linked
bengal graph orphans --level all

Example: Output

📊 Connectivity Distribution
==========================================================================================
  🟢 Well-Connected (≥2.0):      39 pages (31.5%)
  🟡 Adequately Linked (1-2):    38 pages (30.6%)
  🟠 Lightly Linked (0.25-1):    26 pages (21.0%)
  🔴 Isolated (<0.25):           21 pages (16.9%)
==========================================================================================

🟠 Lightly Linked Pages (26 total)
==========================================================================================
#    Score    Path                                          Title
------------------------------------------------------------------------------------------
1    0.50     content/authors/lbliii.md                     Lawrence Lane
2    0.50     content/docs/content/analysis/graph.md        Graph Analysis
3    0.75     content/docs/about/glossary.md                Glossary
4    0.75     content/docs/reference/cheatsheet.md          Cheatsheet
...
==========================================================================================

JSON output (with --format json):

{
  "level_filter": "lightly",
  "distribution": {"isolated": 21, "lightly_linked": 26, ...},
  "pages": [
    {
      "path": "content/docs/about/glossary.md",
      "title": "Glossary",
      "score": 0.75,
      "metrics": {
        "explicit": 0,
        "menu": 0,
        "taxonomy": 0,
        "topical": 1,
        "sequential": 1
      }
    }
  ]
}

Options:

• --level LEVEL: Filter by level - isolated, lightly, adequately, all
• --format FORMAT: Output format - table, json, paths
• --sort FIELD: Sort by path, title, or score
• --limit N: Limit results

Analyze Site Structure

Get an overview of your site's connectivity:

bengal graph analyze

Options:

• --tree: Show site structure as a tree visualization
• --output path/to/graph.html: Generate interactive HTML visualization
• --config PATH: Path to config file (default: bengal.toml)

Short aliases:

bengal g report        # g → graph
bengal analyze         # Top-level alias

PageRank Analysis

Identify your most important pages using Google's PageRank algorithm:

bengal graph pagerank --top-n 10

Example: Output

🏆 Top 10 Pages by PageRank
====================================================================================================
Analyzed 124 pages • Converged in 55 iterations • Damping: 0.85
====================================================================================================
Rank   Title                                         Score        In    Out
----------------------------------------------------------------------------------------------------
1      Template Functions Reference                  0.04669      7.5   2
2      Templating                                    0.03515      6     1
3      Analysis System                               0.02980      2.0   2
4      Health Check System                           0.02592      3.0   2
5      Theme Variables Reference                     0.02559      5.5   2
====================================================================================================

📊 Insights
• Average PageRank score:     0.007300
• Top 10% threshold:          12 pages (score ≥ 0.016534)
• Score concentration:        Moderate

Options:

• --top-n N: Show top N pages (default: 20)
• --damping FLOAT: PageRank damping factor (default: 0.85)
• --format FORMAT: Output format - table, json, csv, or summary

Community Detection

Discover topical clusters in your content:

bengal graph communities --top-n 10 --min-size 3

Options:

• --min-size N: Minimum community size to show (default: 2)
• --resolution FLOAT: Resolution parameter - higher = more communities (default: 1.0)
• --top-n N: Number of communities to show (default: 10)

Bridge Pages Analysis

Find critical navigation pages:

bengal graph bridges --top-n 10

Example: Output

====================================================================================================
🌉 Navigation Path Analysis
====================================================================================================
Analyzed 124 pages • Avg path: 7.05 • Diameter: 19
====================================================================================================

🔗 Top 10 Bridge Pages (Betweenness Centrality)
----------------------------------------------------------------------------------------------------
Rank   Title                                              Betweenness    In    Out
----------------------------------------------------------------------------------------------------
1      Icon Reference                                     0.1419         6.0   5
2      Navigation Directives                              0.1278         3.75  3
3      Object Model                                       0.1142         0.75  2
4      Build Cache                                        0.1117         2.0   2
5      Site Templates Reference                           0.1037         1.0   5

🎯 Top 10 Most Accessible Pages (Closeness Centrality)
----------------------------------------------------------------------------------------------------
Rank   Title                                              Closeness      Out
----------------------------------------------------------------------------------------------------
1      Authors                                            1.0000         1
2      Content Organization                               1.0000         2
3      Template Functions Reference                       1.0000         2
====================================================================================================

📊 Insights
• Average path length:        7.05 hops
• Network diameter:           19 hops
• Structure:                  Deep (consider shortening paths)

Metrics:

• Betweenness: Pages that connect different parts of the site (bridge pages)
• Closeness: Pages easy to reach from anywhere (accessible pages)

Link Suggestions

Get smart recommendations for internal linking:

bengal graph suggest --top-n 50 --min-score 0.5

Options:

• --top-n N: Number of suggestions to show (default: 50)
• --min-score FLOAT: Minimum score threshold (default: 0.3)
• --format FORMAT: Output format - table, json, or markdown

Common Workflows

Improving Internal Linking

1. Get connectivity report:

      bengal graph report
   

2. Find isolated pages (highest priority):

      bengal graph orphans --level isolated
   

3. Find lightly-linked pages (could improve):

      bengal graph orphans --level lightly --format json > lightly-linked.json
   

4. Get link suggestions:

      bengal graph suggest --min-score 0.5 --format markdown > suggestions.md
   

5. Prioritize by importance:

      bengal graph pagerank --top-n 30 --format csv > important-pages.csv
   

CI Integration

Add connectivity checks to your CI pipeline:

# Fail build if too many isolated pages
bengal graph report --ci --threshold-isolated 5

# Export JSON for artifact storage
bengal graph report --format json > connectivity-report.json
bengal graph orphans --format json > under-linked-pages.json

Optimizing Navigation

1. Find bridge pages:

      bengal graph bridges --metric betweenness --top-n 10
   

2. Ensure bridge pages are prominent — Add to main navigation menus.

3. Check accessibility:

      bengal graph bridges --metric closeness --top-n 10
   

Understanding Connectivity Levels

Thereportcommand classifies pages by weighted connectivity:

Why Use Weighted Scores?

Binary orphan detection causes false positives:

• Pages in section hierarchies havetopical links from parent _index.md
• Pages with next/prev navigation havesequentiallinks
• These count as connections but carry low editorial intent

Weighted scores give credit for structural links while highlighting pages that need explicit human-authored links.

Additional Recommendations

Best Practices

Regular Analysis

Run analysis regularly to track improvements:

# Save connectivity report
bengal graph report --format json > analysis-$(date +%Y%m%d).json

# Track isolated pages over time
bengal graph orphans --format json > isolated-$(date +%Y%m%d).json

# PageRank for importance
bengal graph pagerank --format csv > pagerank-$(date +%Y%m%d).csv

Focus on High-Impact Changes

1. Fix isolated pages first (🔴) — Highest priority
2. Improve lightly-linked pages (🟠) — Add explicit cross-references
3. Link to high PageRank pages — Maximum SEO benefit
4. Improve bridge pages — Better navigation
5. Add internal links gradually — Don't over-link

CI/CD Integration

Add connectivity gates to your pipeline:

# GitHub Actions example
- name: Check connectivity
  run: bengal graph report --ci --threshold-isolated 5

Performance Benchmarks

Measured build times for Bengal using the benchmark suite in the repository.

Test Methodology

Test Content Characteristics

The benchmark suite uses minimal test pages:

• Simple Markdown with YAML frontmatter
• No directives (no::: blocks, no {eval-rst})
• No syntax highlighting
• No table of contents generation
• Basic theme with navigation and footer

Important: Real-world sites with directives, syntax highlighting, and complex templates build slower than these benchmarks suggest. See What Slows Builds Down for factors that affect your actual build times.

Build Performance by Site Size

Cold build times (no cache) for the benchmark test content.

These numbers represent best-case performance with minimal content. Your results will vary based on content complexity.

Incremental Build Performance

Time to rebuild when one file changes (warm cache).

Incremental builds provide the largest real-world benefit. Even on large sites, editing a single page rebuilds in under 100 ms.

Memory Usage

Peak RAM during builds with benchmark test content.

Memory per page decreases at scale due to shared overhead (theme, templates, configuration).

Parallel Processing

Build performance with different worker counts (1,000 pages).

Diminishing returns above 4-8 workers due to I/O bottlenecks and coordination overhead.

Free-Threaded Python Impact

Comparing Python 3.14 with and without the GIL (1,000 pages).

# Enable free-threading
PYTHON_GIL=0 bengal build

Kida Template Engine Advantage

Bengal uses Kida for template rendering. Under concurrent workloads, Kida significantly outperforms Jinja2:

This advantage comes from Kida's thread-safe design (copy-on-write updates, local render state, GIL independence). Jinja2 shows negative scaling at high concurrency due to internal contention.

What Slows Builds Down

Real-world sites build slower than benchmarks. Common factors:

Real-World Example: Bengal Documentation Site

The Bengal documentation site itself (the site you're reading) provides a realistic benchmark:

✓ Built 803 pages in 3.19 s (incremental+parallel) | 252.0 pages/sec  # warm caches
✓ Built 803 pages in 6.49 s (incremental+parallel) | 123.6 pages/sec  # cold caches

Even with warm caches, real documentation sites with directive-heavy content show variable performance. The docs use:

• Syntax-highlighted code blocks throughout
• Admonition directives (:::note, :::warning, etc.)
• Tab containers and card grids
• Table of contents on most pages
• Cross-references and backlinks

Realistic Expectations

Your results depend on content complexity. Directive-heavy documentation sites typically see 40-60% of benchmark speeds.

Running Benchmarks

Run the benchmark suite yourself from the repository:

cd benchmarks

# Install dependencies
pip install -r requirements.txt

# Also install bengal in editable mode from project root
pip install -e ..

# Run all benchmarks
pytest -v

# Run specific benchmark
pytest test_build.py -k "incremental" -v

# Run cold build permutations
pytest test_cold_build_permutations.py -v

# Save and compare results
pytest test_build.py --benchmark-save=baseline
pytest test_build.py --benchmark-compare=baseline

Available Benchmark Suites

Performance Tips

1. Use incremental builds during development:bengal build --incremental
2. Enable fast mode:bengal build --fast(quiet output, max speed)
3. Enable template caching: Setcache_templates = truein config (default)
4. Use free-threaded Python:PYTHON_GIL=0with Python 3.14+
5. Minimize directives: Each directive adds parsing and rendering overhead