# `Tptp.Token`
[🔗](https://github.com/jcschuster/tptp/blob/v0.1.3/lib/tptp/token.ex#L1)

The terminal vocabulary of the TPTP grammar.

This module is the single source of truth shared by three consumers:

  * `Tptp.Bnf.Generator`, which resolves the literal text appearing in `::=` rules
    to terminal atoms when it emits `src/tptp_parser.yrl`;
  * `Tptp.Lexer`, which emits these atoms as token categories;
  * `Tptp.Printer.Canonical`, which turns them back into bytes.

A token is a plain three-tuple `{category, offset, length}` — never a struct and
never carrying a binary. `yecc` reads `elem(token, 0)` as the category and
`elem(token, 1)` as the position, so a parse error reports a byte offset
directly. Token text is recovered with `binary_part/3` during the CST post-pass,
which keeps the scan loop free of binary allocation.

Every atom named here is created at compile time. Nothing in this library ever
calls `String.to_atom/1` on input; see `Tptp.Checks.NoDynamicAtoms`, the custom
Credo check that enforces it mechanically.

# `category`

```elixir
@type category() :: atom()
```

A terminal category. Always a compile-time atom.

# `t`

```elixir
@type t() :: {category(), non_neg_integer(), non_neg_integer()}
```

A lexed token: category, byte offset, byte length.

# `categories`

```elixir
@spec categories() :: [category()]
```

Every terminal category, operators and keywords and value categories alike.

# `category_for`

```elixir
@spec category_for(binary()) :: category() | nil
```

The category for a literal spelling as it appears in the BNF, or `nil`.

    iex> Tptp.Token.category_for("~|")
    :nor
    iex> Tptp.Token.category_for("wibble")
    nil

# `dollar_keywords`

```elixir
@spec dollar_keywords() :: [{category(), binary()}]
```

The six `$`-word keywords, which are also legal `<dollar_word>`s.

# `operators`

```elixir
@spec operators() :: [{category(), binary()}]
```

Operators and punctuation only, longest spelling first.

This is what `Tptp.Lexer` generates its scan clauses from. Keywords are excluded
deliberately: a keyword must be recognised by scanning the whole word and then
looking it up, never by matching a prefix, or `filename` lexes as `file` followed
by `name`.

# `punctuation?`

```elixir
@spec punctuation?(category()) :: boolean()
```

Whether a category is pure punctuation, and so dropped from the CST.

    iex> Tptp.Token.punctuation?(:comma)
    true
    iex> Tptp.Token.punctuation?(:ampersand)
    false

# `source_keywords`

```elixir
@spec source_keywords() :: [{category(), binary()}]
```

The four `<source>` keywords, which are also legal `<atomic_word>`s.

# `spelling`

```elixir
@spec spelling(category()) :: binary() | nil
```

The fixed text of a category, or `nil` for a value-carrying category.

    iex> Tptp.Token.spelling(:iff)
    "<=>"
    iex> Tptp.Token.spelling(:lower_word)
    nil

# `spellings`

```elixir
@spec spellings() :: [{category(), binary()}]
```

`{category, spelling}` pairs for every fixed-text terminal, longest spelling first.

Sorted so that a caller matching prefixes in list order gets maximal munch for
free.

# `statement_keywords`

```elixir
@spec statement_keywords() :: [{category(), binary()}]
```

The seven language keywords that open a statement.

# `value_categories`

```elixir
@spec value_categories() :: [category()]
```

Categories whose text must be carried because the category does not determine it.

---

*Consult [api-reference.md](api-reference.md) for complete listing*