org.stekikun.dolmen.debug

Interface TokenVisualizer.LexerInterface<L extends LexBuffer,T,Cat>

Type Parameters:

L - the type of generated lexing buffer to analyze

T - the type of tokens produced

Cat - a category type for tokens

All Superinterfaces:

Tokenizer.LexerInterface<L,T>

Enclosing class:

TokenVisualizer

public static interface TokenVisualizer.LexerInterface<L extends LexBuffer,T,Cat>
extends Tokenizer.LexerInterface<L,T>

This interface acts as a generic proxy to using a Dolmen-generated lexer in the static debugging functions provided in TokenVisualizer. It must provide a way to create a lexer, an entry point to use to tokenize input, and a halting condition used to stop the tokenization (typically, recognizing an end-of-input special token). Additionally, it must provide a way to sort tokens in various categories; tokens in the same category will be highlighted in a similar fashion. The categories can be anything, typical choices could be:

• using Class as the category type and Object.getClass() as the category function, to differentiate between various kinds of valued tokens on one side, and constant tokens on the other side;
• using the token class itself as the category type and Function.identity() as the category function, to differentiate all distinct tokens;
• using the token enum kind generated by a Dolmen parser as the category type, and the corresponding getKind() method to categorize tokens based on what terminal symbol they correspond to.

For convenience, a static factory of(BiFunction, Function, Function, Object) is provided, so that if MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input, one can simply use:

   LexerInterface.of(MyLexer::new, MyLexer::main, Token::getClass, Token.EOF)
 

to build a suitable tokenizing interface for that lexical analyzer which simply differentiates tokens based on their Java class.

Author:

Stéphane Lescuyer

Method Summary

Modifier and Type	Method and Description
Cat	category(T token) Tokens with the same category
static <L extends LexBuffer,T,Cat> TokenVisualizer.LexerInterface<L,T,Cat>	of(BiFunction<String,Reader,L> makeLexer, Function<L,T> entry, Function<T,Cat> categorizer, T eofToken) Typical usage of this method when MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input: LexerInterface.of(MyLexer::new, MyLexer::main, Token::getClass, Token.EOF)

Methods inherited from interface org.stekikun.dolmen.debug.Tokenizer.LexerInterface

entry, halt, makeLexer, of

Method Detail

category

Cat category(T token)

Tokens with the same category

Parameters:

token -

Returns:

the category associated to token

of

static <L extends LexBuffer,T,Cat> TokenVisualizer.LexerInterface<L,T,Cat> of(BiFunction<String,Reader,L> makeLexer,
                                                                              Function<L,T> entry,
                                                                              Function<T,Cat> categorizer,
                                                                              T eofToken)

Typical usage of this method when MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input:

   LexerInterface.of(MyLexer::new, MyLexer::main, Token::getClass, Token.EOF)
 

Parameters:

makeLexer -

entry -

categorizer -

eofToken -

Returns:

a token visualization interface which builds lexical buffers using makeLexer, uses the entry point described by entry, highlights tokens depending on their category as returned by the categorizer, and stops as soon as the token eofToken is encountered.