linkedtok

nimony/src/nifler2/linkedtok.nim

The tree the parser builds, as links rather than as packed tokens.

nifcore.TokenBuf is the right shape for a tree that is read: a flat array of NifToken, a subtree spanned by a jump field, skip an O(1) pointer bump. It is the wrong shape for a tree that is still being rewritten, which is what a parser for Nim's grammar does on almost every node -- a + b has to put the operator in front of an operand that is already written, a declaration with three names fans out into three nodes, a trailing do block moves inside the call in front of it. Against a flat array each of those is a memmove of the region since the mark, and reading the region back means a Cursor, which mints a CursorOwner header for the buffer and frees it again on the next append.

Here a node is a cell: down is its first child, next its next sibling. Wrapping a run of siblings in a new tag, inserting an operator in front of one, moving the last child to the front -- all are pointer assignments that touch a constant number of cells, whatever the size of the subtree.

What does NOT change is the atoms. A payload is a nifcore.NifToken built by nifcore's own constructors, so a tag is still a TagId and a name is still a StrId interned in the same Pool: only the wiring is local to nifler2. The one thing that does not fit in a token's 28-bit payload is a numeric literal, which nifcore spreads over up to three chained tokens; a cell keeps the value in wide instead and flatten hands it back to nifcore's chaining writer untouched.

Cells come from a bump arena. Allocation is a pointer increment, and the whole tree is released by freeing the blocks -- there is nothing per-cell to run, which is what lets the rewrites drop subtrees on the floor instead of splicing them out.

type Node = ptr LinkedToken

type LinkedToken = object
  payload: NifToken
  info: NifLineInfo
  wide: uint64
  down: ptr LinkedToken
  next: ptr LinkedToken

type Arena = object
  blocks: seq[ptr UncheckedArray[LinkedToken]]
  used: int64

proc initArena(): Arena
forces the first alloc to grow
proc destroy(a: var Arena)
Frees every cell at once. Nothing is walked and no cell has a hook to run, so a tree of any shape costs one dealloc per block.
proc cellCount(a: Arena): int64
Cells handed out so far: a size hint for whoever renders the tree.
proc alloc(a: var Arena; payload: NifToken; info: NifLineInfo): ptr LinkedToken
proc allocWide(a: var Arena; k: NifKind; v: uint64; info: NifLineInfo): ptr LinkedToken
A numeric literal: the kind alone in the payload, the bits in wide.
proc kind(n: ptr LinkedToken): NifKind
proc tag(n: ptr LinkedToken): TagId
Only for a TagLit. The jump field of the payload is not maintained here -- down is the structure -- so it is masked out.
proc strId(n: ptr LinkedToken): StrId
Only for an Ident or a StrLit: cells always intern, never inline. identToken/strLitToken put the id in bit 1 upward; bit 0 is the inline/pool-ref flag, and a cell always interns.
proc charVal(n: ptr LinkedToken): char
proc intVal(n: ptr LinkedToken): int64
proc uintVal(n: ptr LinkedToken): uint64
proc floatVal(n: ptr LinkedToken): float64
proc isEmpty(n: ptr LinkedToken): bool
The . that stands for an absent child.
proc childCount(n: ptr LinkedToken): int64
proc copyTree(a: var Arena; n: ptr LinkedToken): ptr LinkedToken
A fresh, unlinked copy of the subtree at n. Needed where one parsed tree ends up in more than one place -- the shared type and value of var a, b: int -- because a cell carries its own next and so can sit in exactly one sibling chain.
proc flatten(dest: var TokenBuf; first: ptr LinkedToken; pool: ref Pool.Obj)
Writes the sibling chain starting at first into dest as ordinary packed tokens, jumps and all -- nifcore's own closeTag computes those. This is the bridge back for everything that reads a TokenBuf: the plugin entry point, and nifler2 tree.