edu.stanford.nlp.trees
Class PennTreebankStreamTokenizer

java.lang.Object
  |
  +--java.io.StreamTokenizer
        |
        +--edu.stanford.nlp.trees.PennTreebankStreamTokenizer

public class PennTreebankStreamTokenizer

extends StreamTokenizer

Builds a tokenizer for PennTreebank (release 2) trees.

Field Summary

Fields inherited from class java.io.StreamTokenizer

nval, sval, TT_EOF, TT_EOL, TT_NUMBER, TT_WORD, ttype

Constructor Summary

PennTreebankStreamTokenizer(Reader r)
Create a tokenizer for PennTreebank trees This sets up all the character meanings for treebank trees

Methods inherited from class java.io.StreamTokenizer

commentChar, eolIsSignificant, lineno, lowerCaseMode, nextToken, ordinaryChar, ordinaryChars, parseNumbers, pushBack, quoteChar, resetSyntax, slashSlashComments, slashStarComments, toString, whitespaceChars, wordChars

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

PennTreebankStreamTokenizer

public PennTreebankStreamTokenizer(Reader r)

Create a tokenizer for PennTreebank trees This sets up all the character meanings for treebank trees

Parameters:

r - The reader steam