Autodoc System

AST-based Python documentation generator

2 min read 338 words

Bengal includes an automatic documentation generation system that extracts API documentation from Python source code using AST-based static analysis.

Note

Zero-Import Extraction: Unlike Sphinx, Bengal's autodoc does not import your code. This means it never crashes due to missing dependencies or side effects.

Architecture

The system follows a clean Extractor → Generator → Template pipeline:

flowchart LR Input[Python Source] --> Parser[AST Parser] Parser --> Extractor[PythonExtractor] Extractor --> Model[DocElement Model] Model --> Generator[Doc Generator] Generator --> Templates[Jinja2 Templates] Templates --> Output[Markdown Files] Output --> Build[Bengal Build] Build --> HTML[Final HTML]

Components

Source Analysis (extractors/)

Parses source code intoDocElementobjects.

  • Python: Usesastmodule to parse classes, functions, and type hints.
  • CLI: Extracts commands and options from Click apps.
  • OpenAPI: (Planned) Extracts endpoints from OpenAPI specs.

Unified Data Model (base.py)

DocElementis the universal representation of any documented item.

  • Structure: Name, description, metadata, children
  • Language Agnostic: Used for Python, CLI, and HTTP APIs
  • Serializable: Can be cached to disk

Markdown Generation (generator.py)

Converts models into Markdown using templates.

  • Two-Pass: Generates Markdown first, then Bengal renders it
  • Parallel: Runs extraction and generation concurrently
  • Caching: Smart caching of generated content

Template Safety

Autodoc templates are designed to never fail the build, even with malformed docstrings.

Error Boundaries

If a template section fails, only that section shows an error. The rest of the page renders normally.

Safe Filters

Custom filters likesafe_descriptionhandles escaping and fallback for missing data.

Configuration

Configure autodoc inbengal.toml:

1
2
3
4
5
[autodoc.python]
enabled = true
source_dirs = ["src/mylib"]
output_dir = "content/api"
docstring_style = "google"  # google, numpy, sphinx

CLI Support

We currently support Click applications via configuration.

1
2
3
4
5
# bengal.toml
[autodoc.cli]
enabled = true
app = "myapp.cli:main"
output_dir = "content/cli-reference"

This generates command references including arguments, options, and hierarchy.