lexbase

nimony/lib/std/lexbase.nim

This module implements a base object of a lexer with efficient buffer handling. Only at line endings checks are necessary if the buffer needs refilling.

The design keeps a '\0' sentinel at buf[sentinel], so the scanning hot loop can read buf[pos] without a bounds check and only has to test for the sentinel; refills happen lazily when a line ending is reached. That makes it a good streaming lexer base for memory-constrained / embedded use: the buffer stays a fixed size (it only grows for pathologically long lines) instead of slurping the whole input.

This is a port of Nim's std/lexbase to Nimony. The JS/VM-only refill branches are dropped (Nimony has a single native target).

const EndOfFile: char
end of file marker
const NewLines: set[char]

type BaseLexer = object
  bufpos: int64
  buf: string
  input: ref StreamObj
  lineNumber: int64
  sentinel: int64
  lineStart: int64
  offsetBase: int64
  refillChars: set[char]
  ioError: bool

proc close(L: var BaseLexer)
closes the base lexer. This closes L's associated stream too.
proc handleCR(L: var BaseLexer; pos: int64): int64
Call this if you scanned over '\c' in the buffer; it returns the position to continue the scanning from. pos must be the position of the '\c'.
proc handleLF(L: var BaseLexer; pos: int64): int64
Call this if you scanned over '\L' in the buffer; it returns the position to continue the scanning from. pos must be the position of the '\L'.
proc handleRefillChar(L: var BaseLexer; pos: int64): int64
Call this if a terminator character other than a new line is scanned at pos; it returns the position to continue the scanning from.
proc open(L: var BaseLexer; input: ref StreamObj; bufLen: int64; refillChars: set[char])
inits the BaseLexer with a stream to read from.
proc getColNumber(L: BaseLexer; pos: int64): int64
retrieves the current column.
proc getCurrentLine(L: BaseLexer; marker: bool): string
retrieves the current line.