Classes
TokenType
0
▼
Classification of lexer tokens.
Categories:
- Delimiters: Block, variable, comment markers
…
TokenType
0
▼
Classification of lexer tokens.
Categories:
- Delimiters: Block, variable, comment markers
- Literals: Strings, numbers, booleans
- Identifiers: Names, keywords
- Operators: Arithmetic, comparison, logical
- Punctuation: Parentheses, brackets, dots
- Special: EOF, whitespace, data (raw text)
Token
5
▼
A single token from the lexer.
Token
5
▼
A single token from the lexer.
Attributes
| Name | Type | Description |
|---|---|---|
type |
TokenType
|
Classification of this token |
value |
str
|
The actual text/value of the token |
lineno |
int
|
1-based line number in source |
col_offset |
int
|
0-based column offset in source Immutable by design for thread-safety. |
Methods
Internal Methods 1 ▼
__repr__
0
str
▼
__repr__
0
str
▼
def __repr__(self) -> str
Returns
str