Module

lexers.jinja_sm

Hand-written Jinja2 lexer using state machine approach.

O(n) guaranteed, zero regex, thread-safe.

Jinja2 syntax: {# ... #} - Comments {{ ... }} - Variable expressions (output) {% ... %} - Statements (if, for, block, etc.)

1Class

Classes

JinjaStateMachineLexer 6 ▼
Jinja2 template lexer. Tokenizes Jinja2 template syntax including: - Comments: {# ... #} - Express…

Jinja2 template lexer.

Tokenizes Jinja2 template syntax including:

  • Comments: {# ... #}
  • Expressions: {{ ... }}
  • Statements: {% ... %}
  • Plain text between template tags

Methods

tokenize 2 Iterator[Token] ▼
def tokenize(self, code: str, config: LexerConfig | None = None) -> Iterator[Token]
Parameters
Name Type Description
code —
config — Default:None
Returns
Iterator[Token]
Internal Methods 5 ▼
_find_closing_braces 4 tuple[int, int, int] ▼
Find closing }} and return updated position and line info.
def _find_closing_braces(self, code: str, pos: int, line: int, line_start: int) -> tuple[int, int, int]
Parameters
Name Type Description
code —
pos —
line —
line_start —
Returns
tuple[int, int, int]
_find_closing_percent 4 tuple[int, int, int] ▼
Find closing %} and return updated position and line info.
def _find_closing_percent(self, code: str, pos: int, line: int, line_start: int) -> tuple[int, int, int]
Parameters
Name Type Description
code —
pos —
line —
line_start —
Returns
tuple[int, int, int]
_tokenize_expression 4 Iterator[Token] ▼
Tokenize inside {{ ... }}.
def _tokenize_expression(self, code: str, pos: int, line: int, line_start: int) -> Iterator[Token]
Parameters
Name Type Description
code —
pos —
line —
line_start —
Returns
Iterator[Token]
_tokenize_statement 4 Iterator[Token] ▼
Tokenize inside {% ... %}.
def _tokenize_statement(self, code: str, pos: int, line: int, line_start: int) -> Iterator[Token]
Parameters
Name Type Description
code —
pos —
line —
line_start —
Returns
Iterator[Token]
_tokenize_jinja_content 5 Iterator[Token] ▼
Tokenize Jinja2 content (expressions or statements).
def _tokenize_jinja_content(self, code: str, pos: int, line: int, line_start: int, end_marker: str) -> Iterator[Token]
Parameters
Name Type Description
code —
pos —
line —
line_start —
end_marker —
Returns
Iterator[Token]