Module

`lexers.plaintext_sm`

Plaintext lexer for code blocks without syntax highlighting.

O(n) guaranteed, zero regex, thread-safe. Simply emits the content as TEXT tokens without any highlighting.

Classes

PlaintextStateMachineLexer 1 ▼

Plaintext lexer - no syntax highlighting, just renders as plain text.

tokenize 4 Iterator[Token] ▼

Tokenize plaintext by splitting into whitespace and text chunks.

def tokenize(self, code: str, config: LexerConfig | None = None, *, start: int = 0, end: int | None = None) -> Iterator[Token]

Name	Type	Description
`code`	`—`	The source code to tokenize.
`config`	`—`	Optional lexer configuration. Default:`None`
`start`	`—`	Starting index in the source string. Default:`0`
`end`	`—`	Optional ending index in the source string. Default:`None`

Iterator[Token]