Module

lexers.plaintext_sm

Plaintext lexer for code blocks without syntax highlighting.

O(n) guaranteed, zero regex, thread-safe. Simply emits the content as TEXT tokens without any highlighting.

1Class

Classes

PlaintextStateMachineLexer 1 ▼
Plaintext lexer - no syntax highlighting, just renders as plain text.

Plaintext lexer - no syntax highlighting, just renders as plain text.

Methods

tokenize 2 Iterator[Token] ▼
Tokenize plaintext by splitting into whitespace and text chunks.
def tokenize(self, code: str, config: LexerConfig | None = None) -> Iterator[Token]
Parameters
Name Type Description
code —

The source code to tokenize.
config —

Optional lexer configuration.

Default:None
Returns
Iterator[Token]