Class SourceReader

A SourceReader allows you to read input from some source, either one single document of content or several named or index source documents, in such a way that each character read registers its position in the source as a tuple index-line-column. That is, the main problem it solves is that of calculating the position of each character read by taking into account characters indicating the end-of-line. It also solves the problem of input divided among several documents, as it is usually the case with source code, and it provides a couple of additional features:

• to use some parts of the input as extra annotations by marking them as non visible, so the input can be read as if the annotations were not there, and
• to allow the relationship of parts of the input with identifiers naming "regions", thus making it possible for external tools to identify those parts with ease.

A SourceReader is created using a SourceInput and then SourcePositions can be read from it. Possible interactions with a SourceReader include:

• peek a character, with peek,
• check if a given strings occurs at the beginning of the text in the current document, without skipping it, with startsWith,
• get the current position as a SourcePosition, with getPosition,
• detect if the end of input was reached, with atEndOfInput,
• detect if the end of the current document was reached, with atEndOfDocument,
• skip one or more characters, with skip,
• read some characters from the current document based on a condition, with takeWhile, and
• manipulate "regions", with beginRegion and endRegion.

There also two global elements that can be accessed:

• the array with the names of the documents composing the source, with documentsNames.
• the characters used to indicate the ending of a line in a document, with lineEnders. When the SourceInput is composed of only one document or by several unnamed ones, the special prefix defaultDocumentNamePrefix is used to identify the documents (with a number suffix to differentiate them). This prefix can be accessed as well as a static property of the SourceReader.

When reading from sources with multiple documents of input, skipping moves inside a document until there are no more characters, then an end of document position is reached (a special position just after the last character of that document), and then a new document is started. Regions are reset at the beginning of each document. When the last document has been processed, and the last end of document has been skipped, the end of input is reached (a special position just after all the documents).

A SourceReader also has a special position, UnknownPosition, as a static member of the class, indicating that the position is not known.

Characters from the input are classified as either visible or non visible. Visible characters affect the line and column calculation, and, conversely, non visible characters do not. Characters are marked as visible by skipping over them normally; characters are marked as non visible by silently skip over them. Visibility of the input affect the information that positions may provide. When skipping characters, the EndOfDocument position must be skipped, although there is no character at that position, and thus, cannot be peeked. This position cannot be skipped as non visible, as every input document is known by the user.

Regarding regions, a "region" is some part of the input that has an ID (as a string). It is used in handling automatically generated code. A typical use is to identify parts of code generated by some external tool, in such a way as to link that part with the element generating it through region IDs. Regions are supposed to be nested, so a stack is used, but no check is made on their balance, being the user responsible for the correct pushing and popping of regions. When skipping moves from one source document to the next, regions are reset, as regions are not supposed to cross different documents of the input.

Example

This is a very basic example using all basic operations. A more complex program will use functions to organize the access with a logical structure, and it will also consider different inputs in the source. Just use this example to understand the behavior of operations -- common usage do NOT follow this structure.

 let pos: SourcePosition;
 let str: string;
 const reader = new SourceReader('program { Poner(Verde) }', '\n');
 // ---------------------------------
 // Read a basic Gobstones program
 if (reader.startsWith('program')) {   // ~~> true
   pos = reader.getPosition();         // ~~> (1,1) as a SourcePosition, with no regions
   // ---------------------------------
   // Skip over the first token
   reader.skip('program');             // Move 7 chars forward
   // ---------------------------------
   // Skip whitespaces between tokens
   while (reader.startsWith(' '))      // ~~> true 1 time
     { reader.skip(); }                // Move 1 char forward (' ')
   // ---------------------------------
   // Detect block start
   if (!reader.startsWith('{'))        // ~~> false (function returns true)
     { fail('Block expected'); }
   reader.beginRegion('program-body'); // Push 'program-body' to the region stack
   str = '';
   // ---------------------------------
   // Read block body (includes '{')
   // NOTE: CANNOT use !startsWith('}') instead because
   //       !atEndOfDocument() is REQUIRED to guarantee precondition of peek()
   while (!reader.atEndOfDocument()    // false
       && reader.peek() !== '}') {     document false 15 times
       str += reader.peek();           // '{', ' ', 'P', ... 'd', 'e', ')', ' '
       reader.skip();                  // Move 15 times ahead
   }
   // ---------------------------------
   // Detect block end
   if (reader.atEndOfDocument())         // ~~> false
     { fail('Unclosed document'); }
   // Add '}' to the body
   str += reader.peek();               // ~~> '}'
   pos = reader.getPosition();         // ~~> (1,24) as a SourcePosition,
                                       //     with region 'program-body'
   reader.endRegion();                 // Pop 'program-body' from the region stack
   reader.skip();                      // Move 1 char forward ('}')
   // ---------------------------------
   // Skip whitespaces at the end (none in this example)
   while (reader.startsWith(' '))      // ~~> false
     { reader.skip(); }                // NOT executed
   // ---------------------------------
   // Verify there are no more chars at input
   if (!reader.atEndOfDocument())      // ~~> false (function returns true)
     { fail('Unexpected document chars after program'); }
   reader.skip();                      // Skips end of document,
                                       //  reaching next document or end of input
   // ---------------------------------
   // Verify there are no more input documents
   if (!reader.atEndOfInput())         // ~~> false (function returns true)
     { fail('Unexpected additional inputs'); }
 }
Copy

NOTE: as peek is partial, not working at the end of documents, each of its uses must be done after confirming that atEndOfDocument is false. For that reason document is better to use startsWith to verify if the input starts with some character (or string), when peeking for something specific.

Private Remarks

The implementation of SourceReader keeps:

• an object associating input document names to input document contents, _documents,
• an object associating input document names to visible input document contents, _visibleDocumentContents,
• an array of the keys of that object for sequential access, _documentsNames,
• an index to the current input document in the array of inputs names, _documentIndex,
• an index to the current visible input document in the array of inputs names (because it may be different from the document index), _charIndex,
• the current line and column in the current input document, _line and _column,
• a stack of strings representing the regions' IDs, _regions, and
• the characters used to determine line ends, _lineEnders.

The object of _documents cannot be empty (with no input document), and all the SourceInput forms are converted to Record<string, string> for ease of access. The _charIndex either points to a valid position in an input document, or at the end of an input document, or the end of input was reached (that is, when there are no more input documents to read).

Line and column numbers are adjusted depending on which characters are considered as ending a line, as given by the property _lineEnders, and which characters are considered visible, as indicating by the user through skip. When changing from one document to the next, line and column numbers are reset.

The visible input is conformed by those characters of the input that has been skipped normally. As visible and non visible characters can be interleaved with no restrictions, it is better to keep a copy of the visible parts: characters are copied to the visible inputs attribute when skipped normally. Visible inputs always have a copy of those characters that have been processed as visible; unprocessed characters do not appear (yet) on visible inputs.

This class is tightly coupled with SourcePosition's implementations, because of instances of that class represent different positions in the source inputs kept by a SourceReader. The operations _documentNameAt, _visibleDocumentContentsAt, _fullDocumentContentsAt, _inputFromToIn, _fullInputFromTo and _fullDocumentContentsAt are meant to be used only by SourcePosition, to complete their operations, and so they are grouped as Protected.

The remaining auxiliary operations are meant for internal usage, to provide readability or to avoid code duplication. The auxiliary operation _cloneRegions is needed because each new position produced with getPosition need to have a snapshot of the region stack, and not a mutable reference.