Lexer (internal/lexer)

A hand-written, single-pass scanner.

Token types

def introduces a variable or constant binding (TOKEN_DEFINE); func introduces a method (TOKEN_FUNC). import (TOKEN_IMPORT) is for file imports (import "path.j";); use (TOKEN_USE) is for library imports (use io;). DOT (.) no longer appears in import syntax (paths are strings now) and is reserved for future expression use. Comparison tokens LE, GE, EQ are two-character (<=, >=, ==) and are recognized by a one-character lookahead from <, >, =. RETURN is the keyword behind return [EXPR]; (see grammar.md).

VARREF carries the variable name without the leading $. STRING carries the value with escape sequences already processed and without surrounding quotes.

Whitespace handling

Spaces, tabs and newlines are discarded between tokens; they only ever advance Line / Col for position tracking. There is no indentation-significant mode and no off-side rule. The user-facing consequence is documented in user-guide/syntax.md > Tokens and whitespace; the rule is load-bearing for jennifer fmt, which trusts that re-emitting the token stream with canonical spacing produces a semantically identical program.

The only place whitespace is retained is inside string literals - readString reads byte-by-byte until the closing quote, so a literal space, tab, or even a raw \n between the quotes becomes part of the string value. Escape sequences (\n, \t, …) are the conventional spelling; raw multi-line literals work too but aren’t the canonical form fmt produces.

Comments and blank lines are emitted as trivia tokens (TOKEN_COMMENT_LINE, TOKEN_COMMENT_BLOCK, TOKEN_COMMENT_SHEBANG, TOKEN_BLANK_LINE) so jennifer fmt can round-trip them. The preprocessor and parser strip these tokens at entry; the formatter walks the raw lexer stream. See Comments below.

Position tracking

Every token records Line and Col (both 1-based) and File (the absolute path supplied to TokenizeWithFile, or "" for unattributed input). The advance() helper bumps line on \n and otherwise bumps col. File flows from the token to the AST node (every node embeds pos{File, Line, Col}), so errors raised inside an imported .j still point at the imported file - see Interpreter > Errors and positions.

Keywords

The lexer’s keyword map covers: def func as init const import use return if elseif else while for true false null and or not int float string bool. Anything else lexed as a word stays a TOKEN_IDENT. define is not a keyword and lexes as a plain identifier. div was removed when // took over floor division.

Comments

# ... runs to end of line and emits TOKEN_COMMENT_LINE; the special case of #! on line 1 col 1 emits TOKEN_COMMENT_SHEBANG instead so the formatter can re-emit the shebang verbatim at the file head. /* ... */ emits TOKEN_COMMENT_BLOCK and nests via a depth counter (increment on /*, decrement on */, exit at depth 0). Unterminated nested comments error positionally at the outermost /* so the message points at where the user meant to start.

Each comment token’s Lexeme carries the verbatim source text including the delimiters (# ..., /* ... */, #! ...) so the formatter round-trips byte-for-byte.

Runs of blank lines collapse to one TOKEN_BLANK_LINE per run - matching the style rule “never more than one consecutive blank line”.

# was chosen (over the C/Java // style) so the floor-division operator // is unambiguous and a Jennifer file can begin with a Unix shebang (#!/usr/bin/env -S jennifer run).

Identifier rule

Variable, method, parameter and library names use [A-Za-z]{1,64} only - no digits, no underscores. Constants use a looser form: uppercase chunks separated by single _ characters - [A-Z]+(_[A-Z]+)*. Every _ must be immediately followed by [A-Z], so leading, trailing and consecutive underscores are all rejected.

The lexer reflects this by accepting _ as a continuation character for bare IDENT tokens (so MAX_RETRIES is a single token) but rejecting any identifier that ends with _. The full per-kind rule is then enforced by the parser at each def / use site - variables, methods, parameters, library names and call callees may not contain _; constants may, with the leading-_ case already excluded by isIdentStart. $var references go through a separate lexer path (readVarRef) that still uses the strict letters-only isIdentPart, so $foo_bar lex-errors directly.