Module

tokens

Token and TokenType definitions for the Patitas lexer.

The lexer produces a stream of Token objects that the parser consumes. Each Token has a type, value, and source location.

Thread Safety:

Token is frozen (immutable) and safe to share across threads. TokenType is an enum (inherently immutable).

Performance Note:

Lexer creates SourceLocation at token creation time to avoid property overhead. Tokens constructed manually (e.g. in tests) use lazy creation when _location_cache is None.

2Classes

Classes

TokenType 0 ▼
Token types produced by the lexer. Organized by category for clarity: - Document structure (EOF, B…

Token types produced by the lexer.

Organized by category for clarity:

  • Document structure (EOF, BLANK_LINE)
  • Block elements (headings, code blocks, quotes, lists)
  • Inline elements (text, emphasis, links, images)
  • Directives and roles
Token 15 ▼
A token produced by the lexer. Tokens are the atomic units passed from lexer to parser. Each token…

A token produced by the lexer.

Tokens are the atomic units passed from lexer to parser. Each token has a type, string value, and source location (lazy).

Performance: SourceLocation is created lazily on first access to.location. This avoids allocation overhead for tokens whose location is never read.

Thread Safety: Frozen dataclass ensures immutability for safe sharing. The lazy cache uses idempotent write (safe for concurrent access).

Attributes

Name Type Description
type TokenType

The token type (from TokenType enum)
value str

The raw string value from source
_lineno int

Start line number (1-indexed)
_col int

Start column offset (1-indexed)
_start_offset int

Absolute start position in source
_end_offset int

Absolute end position in source
line_indent int

Pre-computed indent level of the line (spaces, tabs expand to 4). Set by lexer at token creation; -1 if not computed.
_end_lineno int | None

End line number (for multi-line tokens)
_end_col int | None

End column offset
_source_file str | None

Optional source file path
_location_cache SourceLocation | None —

Methods

location 0 SourceLocation ▼
Get source location (lazily created and cached).
property
def location(self) -> SourceLocation
Returns
SourceLocation SourceLocation object for this token.
lineno 0 int ▼
Line number (convenience accessor).
property
def lineno(self) -> int
Returns
int
col 0 int ▼
Column offset (convenience accessor).
property
def col(self) -> int
Returns
int
Internal Methods 1 ▼
__repr__ 0 str ▼
Compact repr for debugging.
def __repr__(self) -> str
Returns
str