edu.stanford.nlp.trees
Class PennTagbankStreamTokenizer

java.lang.Object
  |
  +--java.io.StreamTokenizer
        |
        +--edu.stanford.nlp.trees.PennTagbankStreamTokenizer

Direct Known Subclasses:

AdwaitStreamTokenizer

public class PennTagbankStreamTokenizer

extends StreamTokenizer

Builds a tokenizer for Penn pos tagged directories.

Field Summary

Fields inherited from class java.io.StreamTokenizer

nval, sval, TT_EOF, TT_EOL, TT_NUMBER, TT_WORD, ttype

Constructor Summary

PennTagbankStreamTokenizer(Reader r)
Create a tokenizer for PennTreebank trees.

Methods inherited from class java.io.StreamTokenizer

commentChar, eolIsSignificant, lineno, lowerCaseMode, nextToken, ordinaryChar, ordinaryChars, parseNumbers, pushBack, quoteChar, resetSyntax, slashSlashComments, slashStarComments, toString, whitespaceChars, wordChars

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PennTagbankStreamTokenizer

public PennTagbankStreamTokenizer(Reader r)

Create a tokenizer for PennTreebank trees. This sets up all the character meanings for treebank trees.

Parameters:

r - The reader steam