nifpools

nimony/src/lib/nifpools.nim

nifpools — nifcore plus the process-global literal/tag pools and the NifLineInfo bridge that nimsem/hexer are architected around. This is the module the frontend imports (usually via nifprelude); lengc-style code that manages its own pools imports nifcore/nifcoreparse directly. The Nim compiler's IC modules use the classic surface in nifstreams.nim instead.

Two model bridges do the real work:

1. Global pool. The frontend is architected around ONE global pool (every intern table) + globalTags + lineMan; nifcore scopes pools per-TokenBuf. We bridge by threading the globals through every createTokenBuf, so ids are comparable across buffers. nifcore's Pool.strings/syms/filenames are plain BiTable[_,string]s, so pool.strings[id] is direct field access; symbols go through the sym* accessors in nifcore (#2457).

2. Line info. The frontend speaks nifcore's NifLineInfo struct directly: .info reads decode the token's LineInfoLit suffix, the info-carrying builders re-attach it via appendLineInfo. The classic PackedLineInfo/lineMan bridge is gone from the frontend; it survives only in nifstreams.nim for the frozen Nim-compiler side. NoLineInfo is kept as the frontend's name for NoNifLineInfo.

Inline literals need no bridge: nifcore stores int/uint/float inline and code reads intVal/uintVal/floatVal directly.

The rest of the module is real API on top of nifcore: info-carrying builders (addParLe/buildTree/copyInto/add*Lit), safe node predicates (isTagLit & friends), raw-token accessors for the CF listing (int28Token/getInt28, symId/strId/tagId on NifToken), skip intents for the pass validator, and parse/render entry points.

var globalTags: ref TagPool.Obj
const NoLineInfo: NifLineInfo
The frontend's classic name for "no line info". A plain NifLineInfo with NoFile; test with isValid, don't compare against it.
var pool: ref Pool.Obj
proc createTokenBuf(cap: int64): TokenBuf
Every shim buffer shares the one global literals + tag namespace, so ids are comparable across buffers (the nifstreams global-pool invariant).
proc initTokenBuf(): TokenBuf
createTokenBuf without the eager storage allocation: the buffer is bound to the global pools, the first add allocates. Use it wherever a buffer used to be left as default(TokenBuf) — an object field, a seq slot — and would otherwise reach the builders with no tag pool at all.
proc registerTag(tag: string): TagId
const ErrT: TagId
the (err …) tag id (classic nifstreams.ErrT)
proc typebits(n: NifToken): int64
Bit width stored in an int-literal type token (classic nifstreams had this). nifcore keeps small ints inline, so the value is the signed operand.
proc info(c: Cursor): NifLineInfo
proc endInfo(c: Cursor): NifLineInfo
proc addStrLit(dest: var TokenBuf; s: StrId; info: NifLineInfo)
proc addStrLit(dest: var TokenBuf; s: string; info: NifLineInfo)
proc isTagLit(c: Cursor): bool
True when the cursor is inside its scope and sits on a tag (compound) node — the nifcore-safe replacement for the classic n.kind == ParLe.
proc isDotToken(c: Cursor): bool
proc isIdent(c: Cursor): bool
proc isSymbol(c: Cursor): bool
proc isSymbolDef(c: Cursor): bool
proc isStringLit(c: Cursor): bool
proc isIntLit(c: Cursor): bool
proc isUIntLit(c: Cursor): bool
proc isFloatLit(c: Cursor): bool
proc isCharLit(c: Cursor): bool
Build-agnostic token predicates (also defined in nifcursors_classic): the sem port uses these instead of n.kind == ParLe/DotToken/… so one source compiles under both the classic and the -d:useNifcore (nifcore) builds.
const InlineInt: NifKind
proc int28Token(operand: int32; info: NifLineInfo): NifToken
CF goto/jump carrier: a DotToken with a 28-bit signed inline payload. It must NOT be an ExtendedSuffix: cursor tokenWidth greedily counts consecutive suffix tokens, so an ExtendedSuffix goto would be glued to whatever token precedes it (or to the previous goto) and every cursor walk over the CF stream would misalign. A DotToken is always its own one-token value; a REAL DotToken has payload 0, so CF walkers treat a zero getInt28 as the plain no-op dot. soperand sign-extends the payload's top bit on read.
proc getInt28(n: NifToken): int32
proc getInt28(c: Cursor): int32
proc charLit(n: NifToken): char
Safe on a raw token: CharLit has no inline/pool duality.
proc tagId(n: NifToken): TagId
proc skipUntilEnd(c: var Cursor)
Skip a bounded cursor's remaining children (stops at the virtual close).
proc isLastSon(n: Cursor): bool
True if n is the last child in its bounded scope.
proc patchInt28Token(n: var NifToken; operand: int32)
proc setSymId(dest: var NifToken; sym: SymId)
Rewrite a Symbol/SymbolDef token's id in place (kind preserved). Goes through internedSymToken so a short name lands in the inline encoding the builders would have produced, rather than a pool ref only this path emits.
template buildTree(dest: var TokenBuf; tag: TagId; info: NifLineInfo; body: untyped)
template buildTree(dest: var TokenBuf; tag: T; info: NifLineInfo; body: untyped)
proc info(n: NifToken): NifLineInfo
Classic tokens carried their line info inline; a bare 4-byte nifcore token cannot. Reading it back yields NoLineInfo — matching the constructors above, which drop the passed info for the same reason. Callers that need real positions must use the buffer-level add* builders or cursors.
proc widenSealed(dest: var TokenBuf; enclosing: int64; growth: int64)
After an insert/replace grew a sealed scope's contents, widen its jump. Handles only the in-field (non-overflow) case; asserts otherwise.
proc widenEnclosingSealed(dest: var TokenBuf; pos: int64; growth: int64)
Widen the jump of EVERY sealed scope enclosing pos after inserting growth tokens there (classic-name mirror; see nifcursors_classic).
proc cursorToPosition(base: Cursor; c: Cursor): int64
Token distance of c from base (classic took two cursors; nifcore's same-named proc takes a TokenBuf, hence this overload).
proc isTagLit(n: NifToken): bool
proc isDotToken(n: NifToken): bool
proc isIdent(n: NifToken): bool
proc isSymbol(n: NifToken): bool
proc isSymbolDef(n: NifToken): bool
proc isStringLit(n: NifToken): bool
proc isIntLit(n: NifToken): bool
proc isUIntLit(n: NifToken): bool
proc isFloatLit(n: NifToken): bool
proc isCharLit(n: NifToken): bool
proc addParLe(dest: var TokenBuf; tag: TagId; info: NifLineInfo)
proc addParRi(dest: var TokenBuf)
proc addParRi(dest: var TokenBuf; info: NifLineInfo)
proc isUnknownToken(c: Cursor): bool
proc isUnknownToken(n: NifToken): bool
proc addUnstructured(dest: var TokenBuf; c: Cursor)
Copy the remaining forest under c verbatim, preserving each subtree's suffixes (line info). Handles a sequence of top-level trees/atoms.
proc insert(dest: var TokenBuf; src: Cursor; pos: int64)
Insert the single subtree at src into dest at token position pos.
proc addSymUse(dest: var TokenBuf; s: SymId; info: NifLineInfo)
proc addSymDef(dest: var TokenBuf; s: SymId; info: NifLineInfo)
proc addDotToken(dest: var TokenBuf; info: NifLineInfo)
proc addIdent(dest: var TokenBuf; s: StrId; info: NifLineInfo)
proc addIdent(dest: var TokenBuf; s: string; info: NifLineInfo)
proc addIntLit(dest: var TokenBuf; v: int64; info: NifLineInfo)
proc addUIntLit(dest: var TokenBuf; v: uint64; info: NifLineInfo)
proc addFloatLit(dest: var TokenBuf; v: float64; info: NifLineInfo)
proc addCharLit(dest: var TokenBuf; v: char; info: NifLineInfo)
proc add(dest: var TokenBuf; src: TokenBuf)
Append a whole buffer (classic dest.add someTokenBuf).
proc writeFile(b: var TokenBuf; filename: string; mode: FileWriteMode)
Serialize the buffer to a textual .nif module file (nifcore renderer).
proc parseFromFile(filename: string; sizeHint: int64): TokenBuf
Whole-file read (classic nifcursors.parseFromFile) via the nifcore reader.
proc parseFromBuffer(input: string; thisModule: sink string; sizeHint: int64): TokenBuf
Parse NIF text into a buffer sharing the global pool/tags (shim invariant).
proc toString(c: Cursor; produceLineInfo: bool): string
Classic toString(Cursor, produceLineInfo); nifcore's is keyword-arg based. The default matches classic nifcursors (true) — plugin inputs and written NIF files rely on it.
proc toString(b: TokenBuf; produceLineInfo: bool): string
Read-only render (the nifcore renderer wants var for its cursor, but only reads); safe to alias an immutable buffer. Accepts both var and immutable.
template linearScan(n: var Cursor; body: untyped)
template copyInto(dest: var TokenBuf; tag: TagId; info: NifLineInfo; body: untyped)
template copyIntoUnchecked(dest: var TokenBuf; tag: string; info: NifLineInfo; body: untyped)
proc takeTree(dest: var TokenBuf; n: var Cursor)

type SkipIntent = enum
  SkipTag = (0, "SkipTag")
  SkipParRi = (1, "SkipParRi")
  SkipName = (2, "SkipName")
  SkipExport = (3, "SkipExport")
  SkipPragmas = (4, "SkipPragmas")
  SkipType = (5, "SkipType")
  SkipExpr = (6, "SkipExpr")
  SkipStmt = (7, "SkipStmt")
  SkipValue = (8, "SkipValue")
  SkipGenParams = (9, "SkipGenParams")
  SkipCond = (10, "SkipCond")
  SkipBody = (11, "SkipBody")
  SkipEffects = (12, "SkipEffects")
  SkipResult = (13, "SkipResult")
  SkipFull = (14, "SkipFull")

func dollar`.SkipIntent(e: SkipIntent): string
template skip(c: var Cursor; intent: SkipIntent)
The intent is documentation only in the nifcore port (the classic runtime predicate depended on the ParLe/ParRi model). skip advances one subtree.
template inc(c: var Cursor; intent: SkipIntent)
As skip(c, intent) but advances past the head only (inc).

type TagClass = enum
  Anything = (0, "Anything")
  AnyExpr = (1, "AnyExpr")
  AnyStmt = (2, "AnyStmt")
  AnyType = (3, "AnyType")

func dollar`.TagClass(e: TagClass): string
template skip(c: var Cursor; expected: TagClass)
Categorical skip intent — documentation only in the nifcore port.
template inc(c: var Cursor; expected: TagClass)
proc consumeParRi(c: var Cursor)
nifcore never materialises a close token — the scope end is implicit at rem == 0. There is nothing to advance past; this exists so classic skipParRi sites compile and stay no-ops.
proc uoperand(c: Cursor): uint32
proc soperand(c: Cursor): int32