org.stekikun.dolmen.debug

Class TokenVisualizer

java.lang.Object

org.stekikun.dolmen.debug.TokenVisualizer

public final class TokenVisualizer
extends Object

This class provides utility methods to help debug lexical analyzers which have been generated with Dolmen. They provide a way to show the exact result of tokenizing some input as a decorated HTML page showing the various tokens and their associated locations on top of the original input contents.

These methods are particularly useful for conveniently checking the behaviour of a lexical analyzer, in particular in terms of the positions associated to each token. These positions are often used in parsers built on top of the lexical analyzer, and are hard to thoroughly check and debug. Visualizing all tokens at once in situ is a great help for that purpose. If you are interested in recording the precise results of a lexical analysis, for instance for the purpose of continuous integration, consider using the Tokenizer debugging feature.

Beware: the token visualization feature does not support lexical analyzers which manipulate the token locations (LexBuffer.curLoc and LexBuffer.startLoc) in ways that are not monotonic. This is for example the case with lexical analyzers supporting GCC-like #line or #include directives, or any analyzer which will move to another input stream midway through the analysis in general. To debug such lexers, consider using the less fancy Tokenizer interface.

Author:

Stéphane Lescuyer

See Also:

visualize(LexerInterface, String, Reader, Writer), string(LexerInterface, String, Writer), file(LexerInterface, String, String)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static interface	TokenVisualizer.LexerInterface<L extends LexBuffer,T,Cat> This interface acts as a generic proxy to using a Dolmen-generated lexer in the static debugging functions provided in TokenVisualizer.

Constructor Summary

Constructors

Constructor and Description
TokenVisualizer()

Method Summary

Modifier and Type	Method and Description
static <L extends LexBuffer,T,Cat> void	file(TokenVisualizer.LexerInterface<L,T,Cat> lexer, String input, String output) Outputs to the given output file a stand-alone HTML page which displays the tokenization of the given input file contents.
static <L extends LexBuffer,T,Cat> void	string(TokenVisualizer.LexerInterface<L,T,Cat> lexer, String input, Writer output) Outputs to the given writer a stand-alone HTML page which displays the tokenization of the given input string's contents.
static <L extends LexBuffer,T,Cat> void	visualize(TokenVisualizer.LexerInterface<L,T,Cat> lexer, String inputName, Reader input, Writer output) Outputs to the given writer a stand-alone HTML page which displays the tokenization of the given input contents.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenVisualizer

public TokenVisualizer()

Method Detail

visualize

public static <L extends LexBuffer,T,Cat> void visualize(TokenVisualizer.LexerInterface<L,T,Cat> lexer,
                                                         String inputName,
                                                         Reader input,
                                                         Writer output)
                                                  throws IOException

Outputs to the given writer a stand-alone HTML page which displays the tokenization of the given input contents. The tokenization is performed based on the lexing interface described by lexer.

Parameters:

lexer - the lexing interface used to tokenize and highlight

inputName - the name of the input stream

input - the input stream to read from

output - the output stream to write the HTML page to

Throws:

IOException

string

public static <L extends LexBuffer,T,Cat> void string(TokenVisualizer.LexerInterface<L,T,Cat> lexer,
                                                      String input,
                                                      Writer output)
                                               throws IOException

Outputs to the given writer a stand-alone HTML page which displays the tokenization of the given input string's contents. The tokenization is performed based on the lexing interface described by lexer.

Parameters:

lexer - the lexing interface used to tokenize and highlight

input - the input string to tokenize

output - the output stream to write the HTML page to

Throws:

IOException

file

public static <L extends LexBuffer,T,Cat> void file(TokenVisualizer.LexerInterface<L,T,Cat> lexer,
                                                    String input,
                                                    String output)
                                             throws IOException

Outputs to the given output file a stand-alone HTML page which displays the tokenization of the given input file contents. The tokenization is performed based on the lexing interface described by lexer.

Parameters:

lexer - the lexing interface used to tokenize and highlight

input - the name of the input file

output - the name of the output file to write the HTML page to

Throws:

IOException