Module

lexers.toml_sm

Hand-written TOML lexer using composable scanner mixins.

O(n) guaranteed, zero regex, thread-safe.

1Class

Classes

TomlStateMachineLexer 2 ▼
TOML lexer using composable mixins.

TOML lexer using composable mixins.

Methods

tokenize 4 Iterator[Token] ▼
def tokenize(self, code: str, config: LexerConfig | None = None, *, start: int = 0, end: int | None = None) -> Iterator[Token]
Parameters
Name Type Description
code —
config — Default:None
start — Default:0
end — Default:None
Returns
Iterator[Token]
Internal Methods 1 ▼
_scan_toml_number 2 tuple[TokenType, int] ▼
Scan TOML number (including dates).
def _scan_toml_number(self, code: str, pos: int) -> tuple[TokenType, int]
Parameters
Name Type Description
code —
pos —
Returns
tuple[TokenType, int]