edu.stanford.nlp.process
Class AbstractTokenizer

java.lang.Object
  |
  +--edu.stanford.nlp.process.AbstractTokenizer

All Implemented Interfaces:

Tokenizer

Direct Known Subclasses:

PTBTokenizer

public abstract class AbstractTokenizer

extends Object

implements Tokenizer

Abstract tokenizer. Tokenizers extending AbstractTokenizer need only implement tokenize(Reader textReader)

Constructor Summary

AbstractTokenizer()

Method Summary

Collection	tokenize(InputStream textInputStream) Returns text as a Collection of Words
abstract Collection	tokenize(Reader textReader) Returns text as a Collection of Words
Collection	tokenize(String text) Returns text as a Collection of Words.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractTokenizer

public AbstractTokenizer()

Method Detail

tokenize

public Collection tokenize(String text)

Returns text as a Collection of Words.

Specified by:

tokenize in interface Tokenizer

tokenize

public abstract Collection tokenize(Reader textReader)

Returns text as a Collection of Words

Specified by:

tokenize in interface Tokenizer

tokenize

public Collection tokenize(InputStream textInputStream)

Returns text as a Collection of Words

Specified by:

tokenize in interface Tokenizer