lexers.python_sm

Hand-written Python lexer using state machine approach.

O(n) guaranteed, zero regex, thread-safe.

Design Philosophy:

This is the reference implementation for Rosettes lexers. It demonstrates:

1. State Machine Architecture: Character-by-character processing with explicit state (position, line, column) as local variables.

2. Frozen Lookup Tables: Keywords, builtins, and operators as frozensets for O(1) membership testing and thread-safety.

3. Fast Path / Slow Path: Simple cases (identifiers, operators) handled inline; complex cases (strings, numbers) delegated to helper methods.

Architecture:

Main Loop (tokenize):

pos = 0
while pos < length:
    char = code[pos]
    # Dispatch based on first character
    if char is whitespace: ...
    elif char is comment: ...
    elif char is string: ...
    # etc.

Helper Methods:

• _scan_string_literal(): Handles prefixed and triple-quoted strings
• _scan_number(): Handles int, float, hex, octal, binary, complex
• _classify_word(): Maps identifiers to KEYWORD, BUILTIN, NAME

Python Language Support:

• All Python 3.x syntax including 3.14
• F-strings (prefix detection)
• Type hints (annotations)
• Walrus operator (:=)
• Match/case statements (3.10+)
• Type parameter syntax (3.12+)
• Unicode identifiers (PEP 3131)

Performance:

• ~50µs per 100-line file
• O(n) guaranteed (single pass, no backtracking)
• ~500 tokens/ms throughput

Thread-Safety:

All state is local totokenize(). Class attributes are frozen:

• _KEYWORDS: frozenset
• _BUILTINS: frozenset
• _TWO_CHAR_OPS: frozenset
• etc.

Adding New Lexers:

Use this file as a template. Key patterns to follow:

1. Frozen lookup tables as module constants
2. Local variables for all state (pos, line, col)
3. Character-by-character dispatch in main loop
4. Helper methods for complex constructs

See Also:

• rosettes.lexers._state_machine: Base class and helper functions
• rosettes._registry: How lexers are registered