nifreader

nimony/src/lib/nifreader.nim

• const StringLit: NifKind

  The reader's traditional name for the string-literal token kind.

• type FilePos = object
    col: int32
    line: int32

• type SymbolPart = enum
    SymDisamb = (0, "SymDisamb")
    SymDedup = (1, "SymDedup")
    SymModule = (2, "SymModule")

• func dollar`.SymbolPart(e: SymbolPart): string
• func dollar`.TokenFlag(e: TokenFlag): string

• type ExpandedToken = object
    tk: NifKind
    flags: set[TokenFlag]
    part: SymbolPart
    suffixes: uint8
    kind: uint16
    data: StringView
    pos: FilePos
    filename: StringView
    comment: StringView

• type Reader = object
    p: ptr UncheckedArray[char]
    eof: pointer
    f: VfsBlob
    buf: string
    thisModule: string
    line: int32
    indexAt: int64
    unusedNameHint: ExpandedToken
    splitSyms: bool
    pending: StringView
    pendingLeft: int32
    pendingTotal: int32
    pendingFlags: set[TokenFlag]
    pendingHasDisamb: bool
    pendingHasModule: bool

• proc $(t: ExpandedToken): string
• proc close(r: var Reader)

  Release the backend's read handle (mmap unmap, LMDB read txn close, …) via the explicit VfsBlob close API.

• proc decodeChar(t: ExpandedToken): char
• proc decodeStr(r: Reader; t: ExpandedToken): string
• proc needsDecoding(t: ExpandedToken): bool

  Whether decodeStr would do anything but copy t.data: an escape to expand, or a module suffix to append. When it is false the bytes ARE the value, so a client that interns them (getOrInclFromView) can skip building a string at all.

• proc decodeComment(t: ExpandedToken): string

  Decode the captured #…# comment, expanding \HH escapes. Returns "" if no comment is attached.

• proc decodeFilename(t: ExpandedToken): string
• proc decodeFloat(t: ExpandedToken): float64
• proc decodeUInt(t: ExpandedToken): uint64
• proc decodeInt(t: ExpandedToken): int64
• proc decodeDisamb(t: ExpandedToken): int64

  The value of a SymDisamb component: abc.12.mod gives 12. Only the leading digits are read (a well-formed symbol has nothing else there).

• proc splitSymbols(r: var Reader; enable: bool)

  Turn the split-symbol parsing mode on or off (default: off).

  A NIF symbol is not a string but an object: <name>.<disamb>.<module>, or <name>.<disamb>.<dedup>.<module> for a generic instantiation, or just <name>.<disamb> for a local symbol. Handing the whole thing to a client as one string is what makes the rest of the compiler take it apart again, over and over.

  In split mode the reader takes it apart ONCE, while it still has the raw bytes: a Symbol/SymbolDef token carries only <name>, and its suffixes field says how many ExtendedSuffix tokens follow — one per remaining component, in order, each tagged with its part. So abc.12.Ikey.mod arrives as

  Symbol "abc" (suffixes: 3), SymDisamb "12", SymDedup "Ikey", SymModule "mod"

  ExtendedSuffix is the binary model's "more bits for the token before me" kind; a symbol component is exactly that, so no new token kind is needed and every existing case over NifKind still compiles.

  An atom with no dot to split on at all -- an operator definition such as []= -- is left whole: suffixes is 0 and the token is what it always was. A symbol whose disambiguator is not a number (arkham's legacy _exit.sys.mod) still yields its module; only the name/disambiguator boundary needs the digit, so everything but the module is the name.

• proc symbolsAreSplit(r: Reader): bool
• proc next(r: var Reader; result: var ExpandedToken)
• proc next(r: var Reader): ExpandedToken

• type DirectivesResult = enum
    WrongHeader = (0, "WrongHeader")
    WrongMeta = (1, "WrongMeta")
    Success = (2, "Success")

• func dollar`.DirectivesResult(e: DirectivesResult): string
• proc startsWith(r: Reader; prefix: string): bool
• proc extractModuleSuffix(filename: string): string
• proc open(filename: string): Reader
• proc openFromBuffer(buf: sink string; thisModule: sink string): Reader

  The Reader keeps buf alive as the owner of the source bytes. buf may be a short SSO string whose chars live inline in the string object; a plain readRawData pointer into it would dangle the moment the Reader is moved into its caller (and into the wrapping Stream), since the inline bytes move with the object. readRawDataStable pins buf to its heap representation, whose payload address survives those moves, so the cached r.p/r.eof stay valid for as long as the Reader (and thus buf) is alive.

• proc processDirectives(r: var Reader): DirectivesResult
• proc fileSize(r: var Reader): int64
• proc offset(r: var Reader): int64
• proc jumpTo(r: var Reader; offset: int64)
• proc indexStartsAt(r: Reader): int64
• proc firstUnusedName(r: Reader): string

  Returns the symbol supplied by the .unusedname directive, or "".