# core URL: /patitas/api/lexer/core/ Section: lexer Description: State-machine lexer with O(n) guaranteed performance. Implements a window-based approach: scan entire lines, classify, then commit. This eliminates position rewinds and guarantees forward progress. No regex in the hot path. Zero ReDoS vulnerability by construction. Thread Safety: Lexer instances are single-use. Create one per source string. All state is instance-local; no shared mutable state. --- > For a complete page index, fetch /patitas/llms.txt. Open LLM text (/patitas/api/lexer/core/index.txt) Share with AI Ask Claude (https://claude.ai/new?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Flexer%2Fcore%2Findex.txt) Ask ChatGPT (https://chatgpt.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Flexer%2Fcore%2Findex.txt) Ask Gemini (https://gemini.google.com/app?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Flexer%2Fcore%2Findex.txt) Ask Copilot (https://copilot.microsoft.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Flexer%2Fcore%2Findex.txt) Module # `lexer.core` State-machine lexer with O(n) guaranteed performance. Implements a window-based approach: scan entire lines, classify, then commit. This eliminates position rewinds and guarantees forward progress. No regex in the hot path. Zero ReDoS vulnerability by construction. Thread Safety: Lexer instances are single-use. Create one per source string. All state is instance-local; no shared mutable state. 1Class ## Classes `Lexer` 10 ▼ State-machine lexer with O(n) guaranteed performance. Uses a window-based approach for block s… Bases: `HeadingClassifierMixin (/patitas/api/lexer/classifiers/heading/#HeadingClassifierMixin)`, `FenceClassifierMixin (/patitas/api/lexer/classifiers/fence/#FenceClassifierMixin)`, `ThematicClassifierMixin (/patitas/api/lexer/classifiers/thematic/#ThematicClassifierMixin)`, `QuoteClassifierMixin (/patitas/api/lexer/classifiers/quote/#QuoteClassifierMixin)`, `ListClassifierMixin (/patitas/api/lexer/classifiers/list/#ListClassifierMixin)`, `LinkRefClassifierMixin (/patitas/api/lexer/classifiers/link_ref/#LinkRefClassifierMixin)`, `FootnoteClassifierMixin (/patitas/api/lexer/classifiers/footnote/#FootnoteClassifierMixin)`, `HtmlClassifierMixin (/patitas/api/lexer/classifiers/html/#HtmlClassifierMixin)`, `DirectiveClassifierMixin (/patitas/api/lexer/classifiers/directive/#DirectiveClassifierMixin)`, `BlockScannerMixin (/patitas/api/lexer/scanners/block/#BlockScannerMixin)`, `FenceScannerMixin (/patitas/api/lexer/scanners/fence/#FenceScannerMixin)`, `DirectiveScannerMixin (/patitas/api/lexer/scanners/directive/#DirectiveScannerMixin)`, `HtmlScannerMixin (/patitas/api/lexer/scanners/html/#HtmlScannerMixin)` State-machine lexer with O(n) guaranteed performance. ``` Uses a window-based approach for block scanning: 1. Scan to end of line (find window) 2. Classify the line (pure logic, no position changes) 3. Commit position (always advances) This eliminates rewinds and guarantees forward progress. Usage: >>> lexer = Lexer("# Hello ``` World") >>> for token in lexer.tokenize(): ... print(token) Token(ATX_HEADING, '# Hello', 1:1) Token(BLANK_LINE, '', 2:1) Token(PARAGRAPH_LINE, 'World', 3:1) Token(EOF, '', 3:6) ``` Thread Safety: Lexer instances are single-use. Create one per source string. All state is instance-local; no shared mutable state. ``` #### Methods `tokenize` 0 `Iterator[Token (/patitas/api/tokens/#Token)]` ▼ Tokenize source into token stream. `def tokenize(self) -> Iterator[Token]` ##### Returns `Iterator[Token (/patitas/api/tokens/#Token)]` Internal Methods 9 ▼ `__init__` 3 ▼ Initialize lexer with source text. `def __init__(self, source: str, source_file: str | None = None, text_transformer: Callable[[str], str] | None = None) -> None` ##### Parameters Name Type Description `source` `—` Markdown source text `source_file` `—` Optional source file path for error messages Default:`None` `text_transformer` `—` Optional callback to transform plain text lines Default:`None` `_dispatch_mode` 0 `Iterator[Token (/patitas/api/tokens/#Token)]` ▼ Dispatch to appropriate scanner based on current mode. `def _dispatch_mode(self) -> Iterator[Token]` ##### Returns `Iterator[Token (/patitas/api/tokens/#Token)]` `_find_line_end` 0 `int` ▼ Find the end of the current line (position of \n or EOF). Uses str.find for O(… `def _find_line_end(self) -> int` Find the end of the current line (position of \n or EOF). Uses str.find for O(n) with low constant factor (C implementation). ##### Returns `int` Position of newline or end of source. `_calc_indent` 1 `tuple[int, int]` ▼ Calculate indent level and content start position. Spaces count as 1, tabs exp… `def _calc_indent(self, line: str) -> tuple[int, int]` Calculate indent level and content start position. Spaces count as 1, tabs expand to next multiple of 4. ##### Parameters Name Type Description `line` `—` Line content ##### Returns `tuple[int, int]` (indent_spaces, content_start_index) `_expand_tabs` 2 `str` ▼ Expand tabs in text to spaces based on start_col (1-indexed). `def _expand_tabs(self, text: str, start_col: int = 1) -> str` ##### Parameters Name Type Description `text` `—` `start_col` `—` Default:`1` ##### Returns `str` `_commit_to` 1 ▼ Commit position to line_end, consuming newline if present. Sets self._consumed… `def _commit_to(self, line_end: int) -> None` Commit position to line_end, consuming newline if present. Sets self._consumed_newline to indicate if a newline was consumed. Uses optimized string operations instead of character-by-character loop. ##### Parameters Name Type Description `line_end` `—` Position to commit to. `_save_location` 0 ▼ Save current location for O(1) token location creation. Call this at the START… `def _save_location(self) -> None` Save current location for O(1) token location creation. Call this at the START of scanning a line, before any position changes. `_make_token` 6 `Token (/patitas/api/tokens/#Token)` ▼ Create a Token with raw coordinates (lazy SourceLocation). O(1) - uses pre-sav… `def _make_token(self, token_type: TokenType, value: str, start_pos: int, *, start_col: int | None = None, end_pos: int | None = None, line_indent: int = -1) -> Token` Create a Token with raw coordinates (lazy SourceLocation). O(1) - uses pre-saved location from _save_location() call. Avoids SourceLocation allocation until token.location is accessed. ##### Parameters Name Type Description `token_type` `—` The token type. `value` `—` The raw string value. `start_pos` `—` Start position in source. `start_col` `—` Optional column override (1-indexed). Default:`None` `end_pos` `—` Optional end position override. Default:`None` `line_indent` `—` Pre-computed indent level. Default:`-1` ##### Returns `Token (/patitas/api/tokens/#Token)` Token with pre-created SourceLocation (avoids property overhead). `_make_token_at_current` 3 `Token (/patitas/api/tokens/#Token)` ▼ Create a Token at current position (for EOF and similar). `def _make_token_at_current(self, token_type: TokenType, value: str, *, line_indent: int = 0) -> Token` ##### Parameters Name Type Description `token_type` `—` The token type. `value` `—` The raw string value. `line_indent` `—` Pre-computed indent level. Default:`0` ##### Returns `Token (/patitas/api/tokens/#Token)` Token at current position with pre-created SourceLocation.