Module

lexers.csv_sm

Hand-written CSV lexer using state machine approach.

O(n) guaranteed, zero regex, thread-safe.

1Class

Classes

CsvStateMachineLexer 1 ▼
CSV file lexer. Highlights: - Quoted strings (including escaped quotes) - Numbers (integers and fl…

CSV file lexer.

Highlights:

  • Quoted strings (including escaped quotes)
  • Numbers (integers and floats)
  • Delimiters (comma, semicolon, tab)
  • Whitespace

Methods

tokenize 2 Iterator[Token] ▼
def tokenize(self, code: str, config: LexerConfig | None = None) -> Iterator[Token]
Parameters
Name Type Description
code —
config — Default:None
Returns
Iterator[Token]