# tokens URL: /patitas/api/tokens/ Section: api Description: Token and TokenType definitions for the Patitas lexer. The lexer produces a stream of Token objects that the parser consumes. Each Token has a type, value, and source location. Thread Safety: Token is frozen (immutable) and safe to share across threads. TokenType is an enum (inherently immutable). Performance Note: Lexer creates SourceLocation at token creation time to avoid property overhead. Tokens constructed manually (e.g. in tests) use lazy creation when _location_cache is None. --- > For a complete page index, fetch /patitas/llms.txt. Open LLM text (/patitas/api/tokens/index.txt) Share with AI Ask Claude (https://claude.ai/new?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Ftokens%2Findex.txt) Ask ChatGPT (https://chatgpt.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Ftokens%2Findex.txt) Ask Gemini (https://gemini.google.com/app?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Ftokens%2Findex.txt) Ask Copilot (https://copilot.microsoft.com/?q=Please%20help%20me%20understand%20this%20documentation%3A%20%2Fpatitas%2Fapi%2Ftokens%2Findex.txt) Module # `tokens` Token and TokenType definitions for the Patitas lexer. The lexer produces a stream of Token objects that the parser consumes. Each Token has a type, value, and source location. Thread Safety: Token is frozen (immutable) and safe to share across threads. TokenType is an enum (inherently immutable). Performance Note: Lexer creates SourceLocation at token creation time to avoid property overhead. Tokens constructed manually (e.g. in tests) use lazy creation when _location_cache is None. 2Classes ## Classes `TokenType` 0 ▼ Token types produced by the lexer. Organized by category for clarity: - Document structure (EOF, B… Bases: `Enum` Token types produced by the lexer. Organized by category for clarity: - Document structure (EOF, BLANK_LINE) - Block elements (headings, code blocks, quotes, lists) - Inline elements (text, emphasis, links, images) - Directives and roles `Token` 15 ▼ A token produced by the lexer. Tokens are the atomic units passed from lexer to parser. Each token… A token produced by the lexer. Tokens are the atomic units passed from lexer to parser. Each token has a type, string value, and source location (lazy). Performance: SourceLocation is created lazily on first access to`.location`. This avoids allocation overhead for tokens whose location is never read. Thread Safety: Frozen dataclass ensures immutability for safe sharing. The lazy cache uses idempotent write (safe for concurrent access). #### Attributes Name Type Description `type` `TokenType (/patitas/api/tokens/#TokenType)` The token type (from TokenType enum) `value` `str` The raw string value from source `_lineno` `int` Start line number (1-indexed) `_col` `int` Start column offset (1-indexed) `_start_offset` `int` Absolute start position in source `_end_offset` `int` Absolute end position in source `line_indent` `int` Pre-computed indent level of the line (spaces, tabs expand to 4). Set by lexer at token creation; -1 if not computed. `_end_lineno` `int | None` End line number (for multi-line tokens) `_end_col` `int | None` End column offset `_source_file` `str | None` Optional source file path `_location_cache` `SourceLocation (/patitas/api/location/#SourceLocation) | None` — #### Methods `location` 0 `SourceLocation (/patitas/api/location/#SourceLocation)` ▼ Get source location (lazily created and cached). property `def location(self) -> SourceLocation` ##### Returns `SourceLocation (/patitas/api/location/#SourceLocation)` SourceLocation object for this token. `lineno` 0 `int` ▼ Line number (convenience accessor). property `def lineno(self) -> int` ##### Returns `int` `col` 0 `int` ▼ Column offset (convenience accessor). property `def col(self) -> int` ##### Returns `int` Internal Methods 1 ▼ `__repr__` 0 `str` ▼ Compact repr for debugging. `def __repr__(self) -> str` ##### Returns `str`