org.stekikun.dolmen.debug

Interface Tokenizer.LexerInterface<L extends LexBuffer,T>

Type Parameters:

L - the type of LexBuffer that this instance handles

T - the type of tokens returned by the lexer's main entry

All Known Subinterfaces:

TokenVisualizer.LexerInterface<L,T,Cat>

Enclosing class:

Tokenizer

public static interface Tokenizer.LexerInterface<L extends LexBuffer,T>

This interface acts as a generic proxy to using a Dolmen-generated lexer in the static debugging functions provided in Tokenizer. It must provide a way to create a lexer, an entry point to use to tokenize input, and a halting condition used to stop the tokenization (typically, recognizing an end-of-input special token).

For convenience, a static factory of(BiFunction, Function, Object) is provided, so that if MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input, one can simply use:

   LexerInterface.of(MyLexer::new, MyLexer::main, Token.EOF)
 

to build a suitable tokenizing interface for that lexical analyzer.

Author:

Stéphane Lescuyer

Method Summary

Modifier and Type	Method and Description
T	entry(L lexbuf) Calls an entry in the given lexing buffer
boolean	halt(T token)
L	makeLexer(String inputName, Reader reader)
static <L extends LexBuffer,T> Tokenizer.LexerInterface<L,T>	of(BiFunction<String,Reader,L> makeLexer, Function<L,T> entry, T eofToken) Typical usage of this method when MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input: LexerInterface.of(MyLexer::new, MyLexer::main, Token.EOF)

Method Detail

makeLexer

L makeLexer(String inputName,
            Reader reader)

Parameters:

inputName -

reader -

Returns:

a lexical analyzer based on the given input stream

entry

T entry(L lexbuf)
 throws LexBuffer.LexicalError

Calls an entry in the given lexing buffer

Parameters:

lexbuf -

Returns:

the next token returned by the entry

Throws:

LexBuffer.LexicalError

halt

boolean halt(T token)

Parameters:

token - the last token returned by entry(LexBuffer)

Returns:

whether tokenization should stop at this token

of

static <L extends LexBuffer,T> Tokenizer.LexerInterface<L,T> of(BiFunction<String,Reader,L> makeLexer,
                                                                Function<L,T> entry,
                                                                T eofToken)

Typical usage of this method when MyLexer has been generated by Dolmen with a main entry point and uses some Token.EOF token for the end-of-input:

   LexerInterface.of(MyLexer::new, MyLexer::main, Token.EOF)
 

Parameters:

makeLexer -

entry -

eofToken -

Returns:

a tokenization interface which builds lexical buffers using makeLexer, uses the entry point described by entry, and stops as soon as the token eofToken is encountered.