Module

lexers.tree_sm

Hand-written Tree-sitter query lexer using state machine approach.

O(n) guaranteed, zero regex, thread-safe. Tree-sitter is a parser generator tool and incremental parsing library.

1Class

Classes

TreeStateMachineLexer 1 ▼
Tree-sitter query language lexer.

Tree-sitter query language lexer.

Methods

tokenize 2 Iterator[Token] ▼
def tokenize(self, code: str, config: LexerConfig | None = None) -> Iterator[Token]
Parameters
Name Type Description
code —
config — Default:None
Returns
Iterator[Token]