# Token Types

URL: /docs/reference/token-types/
Section: reference
Tags: tokens, reference

--------------------------------------------------------------------------------

# Token Types

Complete reference for the `TokenType` enumeration. Each token type maps to a Pygments-compatible CSS class.

## Keywords

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `KEYWORD` | `.k` | Generic keyword | `if`, `else`, `for` |
| `KEYWORD_CONSTANT` | `.kc` | Constant keyword | `True`, `False`, `None` |
| `KEYWORD_DECLARATION` | `.kd` | Declaration | `let`, `const`, `var` |
| `KEYWORD_NAMESPACE` | `.kn` | Namespace keyword | `import`, `from`, `as` |
| `KEYWORD_PSEUDO` | `.kp` | Pseudo keyword | `self`, `cls` |
| `KEYWORD_RESERVED` | `.kr` | Reserved keyword | `goto` |
| `KEYWORD_TYPE` | `.kt` | Type keyword | `int`, `str`, `bool` |

## Names

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `NAME` | `.n` | Generic name | `variable` |
| `NAME_ATTRIBUTE` | `.na` | Attribute | `.attr` |
| `NAME_BUILTIN` | `.nb` | Built-in name | `print`, `len` |
| `NAME_BUILTIN_PSEUDO` | `.bp` | Pseudo built-in | `self`, `cls` |
| `NAME_CLASS` | `.nc` | Class name | `MyClass` |
| `NAME_CONSTANT` | `.no` | Constant name | `PI`, `MAX_SIZE` |
| `NAME_DECORATOR` | `.nd` | Decorator | `@property` |
| `NAME_ENTITY` | `.ni` | Entity name | HTML entities |
| `NAME_EXCEPTION` | `.ne` | Exception name | `ValueError` |
| `NAME_FUNCTION` | `.nf` | Function name | `my_func` |
| `NAME_FUNCTION_MAGIC` | `.fm` | Magic method | `__init__` |
| `NAME_LABEL` | `.nl` | Label | `goto_label:` |
| `NAME_NAMESPACE` | `.nn` | Namespace | `package.module` |
| `NAME_OTHER` | `.nx` | Other name | Misc identifiers |
| `NAME_PROPERTY` | `.py` | Property | `@property` |
| `NAME_TAG` | `.nt` | Tag name | HTML/XML tags |
| `NAME_VARIABLE` | `.nv` | Variable | `$var` |
| `NAME_VARIABLE_CLASS` | `.vc` | Class variable | `@@var` |
| `NAME_VARIABLE_GLOBAL` | `.vg` | Global variable | `$GLOBAL` |
| `NAME_VARIABLE_INSTANCE` | `.vi` | Instance variable | `@var` |
| `NAME_VARIABLE_MAGIC` | `.vm` | Magic variable | `__name__` |

## Literals

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `LITERAL` | `.l` | Generic literal | |
| `LITERAL_DATE` | `.ld` | Date literal | `2026-01-01` |

## Strings

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `STRING` | `.s` | Generic string | `"hello"` |
| `STRING_AFFIX` | `.sa` | String affix | `f`, `r`, `b` |
| `STRING_BACKTICK` | `.sb` | Backtick string | `` `string` `` |
| `STRING_CHAR` | `.sc` | Character | `'c'` |
| `STRING_DELIMITER` | `.dl` | Delimiter | `"`, `'` |
| `STRING_DOC` | `.sd` | Docstring | `"""doc"""` |
| `STRING_DOUBLE` | `.s2` | Double-quoted | `"string"` |
| `STRING_ESCAPE` | `.se` | Escape sequence | `\n`, `\t` |
| `STRING_HEREDOC` | `.sh` | Heredoc | `<<EOF` |
| `STRING_INTERPOL` | `.si` | Interpolation | `{var}` |
| `STRING_OTHER` | `.sx` | Other string | |
| `STRING_REGEX` | `.sr` | Regex | `/pattern/` |
| `STRING_SINGLE` | `.s1` | Single-quoted | `'string'` |
| `STRING_SYMBOL` | `.ss` | Symbol | `:symbol` |

## Numbers

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `NUMBER` | `.m` | Generic number | `42` |
| `NUMBER_BIN` | `.mb` | Binary | `0b1010` |
| `NUMBER_FLOAT` | `.mf` | Float | `3.14` |
| `NUMBER_HEX` | `.mh` | Hexadecimal | `0xff` |
| `NUMBER_INTEGER` | `.mi` | Integer | `42` |
| `NUMBER_INTEGER_LONG` | `.il` | Long integer | `42L` |
| `NUMBER_OCT` | `.mo` | Octal | `0o755` |

## Operators

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `OPERATOR` | `.o` | Operator | `+`, `-`, `=` |
| `OPERATOR_WORD` | `.ow` | Word operator | `and`, `or`, `not` |

## Punctuation

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `PUNCTUATION` | `.p` | Punctuation | `(`, `)`, `,` |
| `PUNCTUATION_MARKER` | `.pm` | Marker | `;`, `:` |

## Comments

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `COMMENT` | `.c` | Generic comment | |
| `COMMENT_HASHBANG` | `.ch` | Hashbang | `#!/bin/bash` |
| `COMMENT_MULTILINE` | `.cm` | Multi-line | `/* ... */` |
| `COMMENT_PREPROC` | `.cp` | Preprocessor | `#include` |
| `COMMENT_PREPROCFILE` | `.cpf` | Preprocessor file | `<stdio.h>` |
| `COMMENT_SINGLE` | `.c1` | Single-line | `// comment` |
| `COMMENT_SPECIAL` | `.cs` | Special comment | `TODO`, `FIXME` |

## Generic (Diffs)

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `GENERIC` | `.g` | Generic | |
| `GENERIC_DELETED` | `.gd` | Deleted line | `-removed` |
| `GENERIC_EMPH` | `.ge` | Emphasis | `*italic*` |
| `GENERIC_ERROR` | `.gr` | Error | |
| `GENERIC_HEADING` | `.gh` | Heading | `# Heading` |
| `GENERIC_INSERTED` | `.gi` | Inserted line | `+added` |
| `GENERIC_OUTPUT` | `.go` | Output | |
| `GENERIC_PROMPT` | `.gp` | Prompt | `>>>` |
| `GENERIC_STRONG` | `.gs` | Strong | `**bold**` |
| `GENERIC_SUBHEADING` | `.gu` | Subheading | `## Sub` |
| `GENERIC_TRACEBACK` | `.gt` | Traceback | |

## Special

| TokenType | Pygments Class | Description | Example |
|-----------|----------------|-------------|---------|
| `TEXT` | (empty) | Plain text | |
| `WHITESPACE` | `.w` | Whitespace | spaces, tabs |
| `ERROR` | `.err` | Error token | Invalid syntax |
| `OTHER` | `.x` | Other | Unclassified |

---

## Usage

```python
from rosettes import tokenize, TokenType

tokens = tokenize("def foo(): pass", "python")

for token in tokens:
    if token.type == TokenType.KEYWORD:
        print(f"Keyword: {token.value}")
    elif token.type == TokenType.NAME_FUNCTION:
        print(f"Function: {token.value}")
```

---

## Semantic Class Mapping

For semantic CSS classes, each TokenType maps to a readable class name:

| TokenType | Semantic Class |
|-----------|----------------|
| `KEYWORD` | `.syntax-keyword` |
| `NAME_FUNCTION` | `.syntax-function` |
| `NAME_CLASS` | `.syntax-class` |
| `STRING` | `.syntax-string` |
| `NUMBER` | `.syntax-number` |
| `COMMENT` | `.syntax-comment` |

See [[docs/styling/css-classes|CSS Classes]] for the complete mapping.

--------------------------------------------------------------------------------

Metadata:
- Word Count: 1003
- Reading Time: 5 minutes