nifcoreparse

nimony/src/lib/nifcoreparse.nim

NIF text parser producing a nifcore.TokenBuf.

Reuses the existing nifreader, whose ExpandedToken is independent of the in-memory representation: its ParLe/ParRi events drive nifcore's openTag/closeTag, and atoms drive the builder. The reader's relative line positions are resolved to absolute file/line/col (against a parent stack, exactly like nifstreams.rawNext) and emitted as a sparse LineInfoLit suffix only where the position changes. Callers that need random-access effective locations can request dense line info while parsing.

Note: NIF #comment# decorations are not carried into the token buffer — the codegen path has no use for them.

proc peekRootInfo(r: var Reader; pool: ref Pool.Obj): NifLineInfo
Absolute line info of the next token — typically a module's toplevel (stmts) head, which carries the file explicitly — without consuming it. Index-jumped decl reads pass this as parse's parentSeed so their file-less child position deltas resolve against the module root (a decl's parent in the original tree IS the toplevel (stmts) node).
proc parse(r: var Reader; b: var TokenBuf; parentSeed: NifLineInfo; denseLineInfo: bool)
Read one complete NIF tree (or until EOF) from r into b. parentSeed seeds the parent line-info stack so the first token's relative position resolves against the right origin (index-jumped reads pass the indexed compound's parent info; whole-file reads pass NoNifLineInfo). With denseLineInfo, every positioned value receives its effective location instead of only changes in the location stream.
proc parseFromBuffer(input: string; thisModule: sink string; sizeHint: int64; sharedPool: ref Pool.Obj; sharedTags: ref TagPool.Obj; denseLineInfo: bool): TokenBuf
proc parseFromBuffer(input: string; thisModule: sink string; unusedName: var string; sizeHint: int64; sharedPool: ref Pool.Obj; sharedTags: ref TagPool.Obj; denseLineInfo: bool): TokenBuf
Parses NIF text and returns its .unusedname directive via unusedName.
proc parseFromFile(filename: string; sizeHint: int64; sharedPool: ref Pool.Obj; sharedTags: ref TagPool.Obj; denseLineInfo: bool): TokenBuf
proc parseFromFile(filename: string; unusedName: var string; sizeHint: int64; sharedPool: ref Pool.Obj; sharedTags: ref TagPool.Obj; denseLineInfo: bool): TokenBuf
Parses a NIF file and returns its .unusedname directive via unusedName.
proc appendTo(b: var TokenBuf; dest: var Builder; includeLineInfo: bool)
Appends canonical NIF text for the whole buffer to dest.
proc toString(b: var TokenBuf; sizeHint: int64; includeLineInfo: bool): string
Canonical NIF text for the whole buffer (one or more top-level values). Set includeLineInfo to false for location-free diagnostic rendering.
proc toString(node: Cursor; sizeHint: int64; includeLineInfo: bool): string
Canonical NIF text for the value or subtree at node.
proc toModuleString(b: var TokenBuf; dottedSuffix: string; sizeHint: int64; compact: bool): string
Like toString but emits a full module file: the (.nif27) header with a patched .indexat, the body, and a trailing (.index …) mapping each global SymbolDef to its declaration's byte offset. dottedSuffix (e.g. .mymod) compresses self-module symbol suffixes to a trailing dot (the reader re- expands via its thisModule). Mirrors nifstreams.toModuleString so the output is byte-compatible with the canonical tooling (reindex).
compact drops the layout whitespace (nifbuilder's compact mode). The result parses identically — indentation is decoration — and is what a text NIF costs when it is treated as a cache rather than as something to read.