parserrt

nimony/src/nifler2/parserrt.nim

• template grammar(rules: varargs[untyped])

  The grammar notation of doc/internals/parser_generator.md, turned into one pRule proc per rule at compile time.

• type IndClass = enum
    icNoInd = (0, "icNoInd")
    icLt = (1, "icLt")
    icEq = (2, "icEq")
    icGt = (3, "icGt")

• func dollar`.IndClass(e: IndClass): string

• type PrimaryMode = enum
    pmNormal = (0, "pmNormal")
    pmTypeDesc = (1, "pmTypeDesc")
    pmTypeDef = (2, "pmTypeDef")
    pmTrySimple = (3, "pmTrySimple")

• func dollar`.PrimaryMode(e: PrimaryMode): string

• type OptSigs = object
    posMarkers: int64
    kvs: int64

• type Mark = object
    prev: ptr LinkedToken
    info: NifLineInfo
    sigs: OptSigs

• type Parser = object
    lex: Lexer
    tok: Token
    arena: Arena
    first: ptr LinkedToken
    last: ptr LinkedToken
    file: FileId
    currInd: int32
    indStack: seq[int32]
    inPragma: int64
    prevKind: TokKind
    filterFailed: bool
    prevEndLine: int64
    prevEndCol: int64
    inSemiStmtList: int64
    sigs: OptSigs
    sections: seq[NiflerKind]
    lastSection: NiflerKind
    infos: seq[NifLineInfo]
    wrapFields: seq[bool]
    pool: ref Pool.Obj
    tags: ref TagPool.Obj
    failed: bool
    errLine: int64
    errCol: int64
    errMsg: string

• proc openParser(src: string; filename: string; pool: ref Pool.Obj; tags: ref TagPool.Obj): Parser

  pool and tags are the output's: nifler2 writes through nifpools' globals, a plugin parses into its own. tags is not stored -- a cell holds a TagId and the tag pool is only consulted when the tree is written out.

• proc close(p: var Parser)

  Releases the whole tree. One dealloc per arena block -- the cells have no hooks and the chain is not walked.

• proc finish(p: var Parser): TokenBuf

  The parsed tree as packed tokens, for everything that reads a TokenBuf.

• proc info(p: Parser): NifLineInfo
• const errInvalidIndentation: string
• proc prettyTok(t: Token): string

  prettyTok in compiler/lexer.nim.

• proc errorAt(p: var Parser; line: int64; col: int64; msg: string)

  The first syntax error ends the parse. Recovery would mean bailing out of an arbitrarily deep recursion, and the generated code cannot: there are no exceptions in this runtime and no notation for a recovery production. So the error is recorded and the token stream ends here: from now on the current token is an end of file that getTok never moves past. That is what unwinds the recursion -- every repetition's continuation test fails on it, and a mandatory item that does not match reports into an error that is already recorded. It has to end the stream rather than merely record the error, because expect does not consume the token it did not match, so a repetition whose body fails would make no progress and spin. A rule cut short leaves fewer trees than its layout expects, which is why the layouts check failed.

• proc reportFailure(p: Parser)

  nifler's output for a syntax error, and its exit code. The lexer's messages come first, which is where Nim, one token ahead as well, prints them; the stream ended at the error, so they are the ones up to it.

• proc error(p: var Parser; msg: string)

  parMessage: at the current token.

• proc lexError(p: var Parser; msg: string)

  lexMessage: at the lexer's position, the end of the current token.

• proc identExpected(p: var Parser)
• proc exprExpected(p: var Parser)
• proc ruleError(p: var Parser; rule: string; misplaced: bool)

  Nothing in rule starts with the current token. parser.nim has no such single place: the message is whichever check of the hand-written procedure the token trips first. For a token that could start the rule at another indentation (misplaced) that is usually an indentation check, and otherwise identOrLiteral's errExprExpected.

• proc strictListEnd(p: var Parser)

  See strictListEnd in the grammar.

• proc strictListStart(p: var Parser)

  An indented token that cannot start the list at all.

• proc enumListEnd(p: var Parser)

  parseEnum loops on validInd and calls parseSymbol on whatever is there.

• proc paramStart(p: var Parser)

  parseParamList's messages for a token that starts no parameter.

• proc accentEnd(p: var Parser)
• proc listEnd(p: var Parser; close: TokKind)

  See listEnd in the grammar. A pragma ends in .} or }.

• proc missingEquals(p: var Parser)

  A routine without a body followed by an indented line.

• proc requireFields(p: var Parser; m: Mark)
• proc funcType(p: var Parser)
• proc stmtListEnd(p: var Parser)

  parseStmt's block loop ends on these; any other token at the block's indentation is handed to complexOrSimpleStmt, which finds nothing.

• proc requireExcept(p: var Parser; m: Mark)
• proc noIndHere(p: var Parser)
• proc blockNameEnd(p: var Parser)
• proc tupleEnd(p: var Parser)
• proc indentError(p: var Parser)
• proc getTok(p: var Parser)
• proc expect(p: var Parser; k: TokKind)
• proc expect(p: var Parser; k: TokKind; s: string)

  The spelling matters for the operators the grammar names literally ('->' in paramListArrow).

• proc expectLeaf(p: var Parser; k: TokKind)

  @'not' in the grammar: the terminal is content, not punctuation.

• proc expectLeaf(p: var Parser; k: TokKind; s: string)
• proc indClass(p: Parser): IndClass
• proc checkInd(p: var Parser; allowed: set[IndClass])
• proc afterOperator(p: var Parser)

  simpleExprAux after it consumed a binary operator: flexComment and optPar.

• proc pushInd(p: var Parser)
• proc pushIndAny(p: var Parser)

  withInd without the realInd assertion: the indentation becomes the current token's whatever that is, including -1 for a token that is not first on its line. semiStmtList is the one place parser.nim does this, and it is why (\n when a: x\n elif b: y) has its elif at the same indentation as its when rather than at the enclosing block's.

• proc popInd(p: var Parser)
• proc noSpaceBefore(p: Parser): bool

  f(x) is a call, f (x) a command.

• proc isUnary(p: Parser): bool
• proc isSigilLike(p: Parser): bool
• proc dotLikeOps(p: Parser): bool

  nimPreviewDotLikeOps: a.?b would parse as a field access rather than as an infix operator. nifler does not define it, so a dot-like operator is an ordinary infix operator -- a.?b.c is (infix .? a (dot b c)) -- and the answer is false.

• proc inTypeDesc(p: Parser; mode: PrimaryMode): bool

  parser.nim's if mode == pmTypeDesc in commandParam.

• proc parIsTuple(p: Parser; mode: PrimaryMode): bool

  parser.nim's identOrLiteral takes '(' to the comma-separated exprColonEqExprList in a type, and to parsePar -- which also accepts a statement list, an assignment and a do block -- everywhere else. Both productions exist in the grammar; this is the discriminator that says which one '(' opens.

• proc inOrOut(p: Parser): bool

  parseGenericParam's of tkIn, tkOut: -- the variance markers of MyPtr[out T]. The grammar spells the operand as KEYW so that the keyword is emitted as the prefix operator's name, which a terminal would not be; this narrows KEYW back to the two that are meant.

• proc pragmaOnPrimary(p: Parser; mode: PrimaryMode): bool

  simpleExprAux's if p.tok.tokType == tkCurlyDotLe and (p.tok.indent < 0 or realInd(p)) and mode == pmNormal. The indentation half is spelled in the grammar; this is the mode half. pmTrySimple counts because simpleExprAux rewrites it to pmNormal right after primary returns, before it looks for the pragma.

• proc isTypedefOperand(p: Parser; mode: PrimaryMode): bool

  parseTypeDescKAux's isTypedef: after ref/ptr/distinct in a type definition, an object or tuple operand brings a whole declaration with it -- type T = ref object and its indented field list -- while anything else is just a primary. It is a two-token decision in parser.nim and a one-token one here, because by the time the operand is dispatched the keyword is already consumed.

• proc typeOperandFollows(p: Parser): bool

  parseTypeDescKAux's not isOperator(p.tok) and isExprStart(p): what makes ptr in SomeInteger | ptr | pointer a bare ptr rather than the prefix of (prefix | pointer). The isExprStart half is the alternative's FIRST set and is already in the generated condition; the operator half is not expressible there, because an operator can also start a prefix expression.

• proc commandStart(p: Parser): bool

  parser.nim's guard on primarySuffix's command branch. The token set is the one its case lists, which is narrower than FIRST(commandParam): not, if, addr and friends can start an expression but not a command, so ref int not nil is (infix not (ref int) nil) rather than a command whose argument is not nil. An infix operator is not the start of a command either -- import std / os is (infix / std os) -- and inside a pragma nothing is, because {.push hints:off.} must not become {.push(hints:off).}.

• proc commandAllowed(p: Parser; mode: PrimaryMode): bool

  parser.nim's commandExpr returns its operand untouched when the mode is pmTrySimple, so in that mode a command is not a suffix at all. That is what leaves echo a, b for exprStmt to parse as one command with two parameters, instead of (cmd echo a) followed by a stray comma.

• proc suffixStart(p: Parser): bool

  parser.nim's primarySuffix loop guard, which the grammar cannot spell: a suffix continues on the same line, and a . may also open a continuation line that is indented at least as far as the line the primary started on. That last indentation is a parameter of primarySuffix there; here it is the enclosing block's, which differs only for a primary that already spans lines.

• proc getPrecedence(p: Parser): int64

  parseOperators loops while opPrec >= limit and p.tok.indent < 0 and not isUnary(p.tok). The unary test lives here rather than in the generator's binary(...), because "a unary operator has no infix precedence" is a fact about Nim's operators and not about precedence climbing. Without it echo $kind, "a" parsed as (infix $ echo kind) and then stalled on the comma.

• proc isRightAssoc(p: Parser): bool
• proc mark(p: Parser): Mark
• proc discardUnused(m: Mark)

  A mark the rule turned out not to need. The generator emits one per alternative because it cannot know, before it has seen the whole alternative, whether a tag will claim it.

• proc append(p: var Parser; n: ptr LinkedToken)

  Links n as the last sibling. A cell carries its own next, so it lives in one chain only: appending unhooks it from wherever it was.

• proc tagId(k: NiflerKind): TagId

  The master tag pool is seeded in TagEnum order, so a tag's id is its ordinal and no string is ever hashed to find it.

• proc wrapAt(p: var Parser; m: Mark; tag: NiflerKind; info: NifLineInfo)

  Makes everything since m the children of a new (tag ...) node. The chain is not touched: it becomes the new node's down, and the node takes its place in the enclosing chain. Constant time, whatever it holds.

• proc wrap(p: var Parser; m: Mark; tag: NiflerKind)

  Retroactively make everything from m a (tag ...) node.

• proc insertTokAt(p: var Parser; m: Mark; k: TokKind)

  ^tag[ @'not' ... ]: consume the operator and insert it at the anchor, in front of the operand that is already there.

• proc insertLeafAt(p: var Parser; m: Mark; text: string)

  binary(...)'s operator: a + b is (infix + a b), so the operator has to land before the left operand, which was parsed before it was read. Two links, where a flat buffer had to move the whole operand out of the way.

• proc insertOpLeafAt(p: var Parser; m: Mark)

  insertLeafAt with the current token's text: p.tok.s passed directly would alias the var Parser.

• proc emitEmpty(p: var Parser)

  . in the grammar: an absent child.

• proc setExportMarker(p: var Parser; m: Mark)

  exportMarker: nifler writes x in the export slot, whatever the operator was.

• proc pushSection(p: var Parser; tag: NiflerKind)
• proc pushLastSection(p: var Parser)

  using: nifler's section is one variable that nothing restores, and nkUsingStmt does not set it -- a using gets the tag of whatever section was opened last in the module. Nothing downstream reads that tag, but a byte-identical tree has to have it.

• proc popSection(p: var Parser)
• proc pushFieldWrap(p: var Parser; wrap: bool)
• proc popFieldWrap(p: var Parser)
• proc wrapLikeFirst(p: var Parser; m: Mark; tag: NiflerKind)

  A node positioned at its first child: newTree(nkCommand, a.info, a).

• proc fanOut(p: var Parser; m: Mark)

  name x pragmas repeated, then type value: one (section name x pragmas type value) per name. The count is exact because every slot is written, . or not -- which is what the placeholders buy.

• proc fanOutKv(p: var Parser; m: Mark)

  A tuple field list: names, then type value. nifler keeps (kv name type) and drops the default.

• proc joinIdents(p: var Parser; m: Mark)

  Inside backquotes parseSymbol glues a run of operator and bracket tokens into one identifier: []= is (quoted []=), not three children, while =copy is (quoted = copy).

• proc inSemiStmtList(p: Parser): bool

  parseStmt's if p.inSemiStmtList > 0: result = simpleStmt(p): inside a parenthesised statement list a one-line body is the bare statement, not a stmts -- (if a: b else: c) is (elif a b). The counter is not reset by what nests inside, so neither is this.

• proc stmtListExprLayout(p: var Parser; m: Mark)

  nkStmtListExpr: nifler writes all statements but the last in a stmts and the last one after it, (expr (stmts a b) c).

• proc rhsMode(mode: PrimaryMode): PrimaryMode

  The mode of an operator's right operand. simpleExprAux turns pmTrySimple into pmNormal after the first primary, and parseOperators turns pmTypeDef into pmTypeDesc -- so x == Handle 0 is (infix == x (cmd Handle 0)).

• proc literalAsIdent(p: var Parser; m: Mark)

  Inside backquotes parseSymbol makes a literal token an identifier of its text: 'big is (quoted ' big).

• proc posMarker(p: var Parser)

  A . that only carries the current position to a layout, which removes it again.

• proc dotLayout(p: var Parser; m: Mark)

  dotExpr's rewrite of x.y[:z](args) into y[z](x, args): the dot node's children are x y (at z...) args... when the [: was there. The rule wrapped one (dot ...) tree, so with no [: there is nothing to move and the tree stands as written.

• proc curlyOrTable(p: var Parser; m: Mark)

  setOrTableConstr retags nkCurly as nkTableConstr as soon as one element is key: value. No kv was built inside: not a table, so the (curly ...) stands.

• proc callOrObjConstr(p: var Parser; m: Mark)

  primarySuffix's (: a call whose first argument is name: value is an object constructor, Foo(a: 1) is (oconstr Foo (kv a 1)). No kv was built inside: not an object constructor, so the (call ...) stands. A kv nested deeper than the first argument only costs the test below, which then finds none in that slot.

• proc attachBlocks(p: var Parser; m: Mark)

  postExprBlocks: a trailing : or do block belongs to the expression in front of it. makeCall keeps a node that is already a call -- foo x: stays a cmd -- and wraps anything else, and every block is appended as a child: c.into: is (call (dot c into) (stmts ...)). The rule that calls this has parsed the operand first, so the operand is the first tree since m and the blocks are the rest.

• proc doLayout(p: var Parser; m: Mark; atBody: bool)

  A do block, parsed as params ret pragmas body. Without a signature or pragmas it is just its body; otherwise (do params ret body) -- nifler writes the formal parameters as (params ...) plus the return type, and drops the pragmas.

• proc routineBodyAllowed(p: Parser; mode: PrimaryMode): bool

  parseProcExpr(p, mode != pmTypeDesc, ...): in a type proc (): int = x has no body -- the = x is the declaration's default value.

• proc emptyDiscriminator(p: var Parser)

  parseObjectCase without a discriminator: an nkIdentDefs of empty nodes. Its name is the empty node, whose position has no file, so bridge.nim writes it absolute -- as ~1,,???.

• proc wrapNoInfo(p: var Parser; m: Mark; tag: NiflerKind)

  A node bridge.nim writes with addTree alone.

• proc pragmaBlock(p: var Parser; m: Mark)

  parseStmtPragma: a pragma followed by a block is an nkPragmaBlock, (pragmax pragmas body), positioned at the pragma.

• proc inheritLayout(p: var Parser; m: Mark)

  nkOfInherit, created at of: one type is written as itself, several as (par A B).

• proc wrapSection(p: var Parser; m: Mark)

  A declaration that is already in slot form, tagged with the section: the lets of a tuple unpacking, which bridge.nim writes without a position.

• proc pushInfo(p: var Parser)

  The position of the current token, for a node built after its keyword is consumed -- parseRoutine creates the routine node at proc.

• proc popInfo(p: var Parser)
• proc moveLastToMark(p: var Parser; m: Mark)

  for x in it and let (a, b) = v: nifler writes the iterated or unpacked value first.

• proc procLayout(p: var Parser; m: Mark; keyword: NiflerKind)

  An anonymous routine, parsed as params ret pragmas body with . for each one that is absent. With a body it is a lambda, (proc . . . . params ret pragmas . body), and (params) is always there. Without one it is a type, and a type keeps what the source had: nothing at all is (proctype), no signature is a single . in place of params and result.

• proc routineLayout(p: var Parser; m: Mark; keyword: NiflerKind)

  A named routine, parsed as name x pattern typevars params ret pragmas body: (keyword name x pattern typevars params ret pragmas . body), with the effects slot nifler reserves and (params) always present.

• proc emitLeaf(p: var Parser)

  The terminal the grammar matched, as a NIF atom. Which atom is decided by the token kind alone, so the generator never has to say (it emits the terminal's name as a comment only).