Class Token

The Token class is a simple class representing the tokens that conform the source code, and it only contains information: the kind of token, the information in the source code that generates the token, and information about the position of that information in the source. In addition to the operations to create it and to access the mentioned information, it has a way to obtain a printable version of the token, and to inform if the token is a "filler". Filler tokens are those used by the Gobstones lexer to fill the space in between structural tokens (such as keywords, identifiers, and numbers), and that are ignored by the parser. Tokens that are not fillers will be called GTokens.

Properties

_implementationDetails

API: Access

span tag value isFiller toString

API: Constructor

constructor

Implementation: Internal state

_span _tag _value

Static operations

lowerid num string symbid symbolAsString upperid

Properties

Static Private _implementationDetails

_implementationDetails: string = 'Dummy for documentation'

The Token class is a simple class representing the tokens that conform the source code, and it only contains information. The internal state has 4 fields for that:

  • _tag, indicating the kind of token this one is,
  • _value, with the information in the source code that generates the token, and
  • _span (_start and _end positions), with information about the span of the value in the source code. It also has functions to produce printable information of the token, and to inform if the token is a "filler" -- see Token for an explanation on "fillers".

API: Access

span

  • get span(): Span

  • The span in the source where this token starts.

    Returns Span

tag

  • get tag(): symbol

  • The tag of the token, indicating the kind of token this one is.

    Returns symbol

value

  • get value(): string

  • The value of the token, with the information in the source code that generates the token.

    Returns string

isFiller

  • isFiller(): boolean

  • Indicates if the token is a filler token. A filler token is one that is used by the Gobstones lexer to fill the space in between structural tokens (such as keywords, identifiers, and numbers), and that is ignored by the parser. Tokens that are not fillers are called GTokens.

    Returns boolean

toString

  • toString(): string

  • A string version of the token, for printing purposes. It is NOT intended for persistance, as it looses information.

    Returns string

API: Constructor

constructor

  • new Token(tag, value, span?): Token

  • A token consists of a tag indicating its kind, a value, with the information in the source code that generates the token, and possibly information about the position of that information in the source code.

    Parameters

    • tag: symbol

      indicating the kind of token to create.

    • value: string

      with the information in the source code that generates the token.

    • span: Span = ...

      the span in the source where this token comes from.

    Returns Token

Implementation: Internal state

Private _span

_span: Span

The span in the source where this token is (start and end positions of it).

Private _tag

_tag: symbol

The tag of the token, indicating the kind of token this one is.

Private _value

_value: string

The value of the token, with the information in the source code that generates the token.

Static operations

Static lowerid

  • lowerid(value, span?): Token

  • Construct an LOWERID Token.

    Parameters

    • value: string
    • Optional span: Span

    Returns Token

Static num

  • num(value, span?): Token

  • Construct an NUM Token.

    Parameters

    • value: string
    • Optional span: Span

    Returns Token

Static string

  • string(value, span?): Token

  • Construct an STRING Token.

    Parameters

    • value: string
    • Optional span: Span

    Returns Token

Static symbid

  • symbid(value, span?): Token

  • Construct an SYMBID Token.

    Parameters

    • value: string
    • Optional span: Span

    Returns Token

Static symbolAsString

  • symbolAsString(s): string

  • Gives the string for a symbol, without the 'Symbol(...)' in it.

    Parameters

    • s: symbol

    Returns string

Static upperid

  • upperid(value, span?): Token

  • Construct an UPPERID Token.

    Parameters

    • value: string
    • Optional span: Span

    Returns Token

Settings

Member Visibility

Theme

On This Page

_implementationDetailsspantagvalueisFillertoStringconstructor_span_tag_valueloweridnumstringsymbidsymbolAsStringupperid