edu.stanford.nlp.process
Interface Tokenizer

All Known Implementing Classes:

AbstractTokenizer, SimpleTokenizer

public interface Tokenizer

A tokenizer has methods for tokenizing a String, Reader, or InputStream as words.

Method Summary

Collection	tokenize(InputStream textInputStream) Returns text as a Collection of Words
Collection	tokenize(Reader textReader) Returns text as a Collection of Words
Collection	tokenize(String text) Returns text as a Collection of Words.

Method Detail

tokenize

public Collection tokenize(String text)

Returns text as a Collection of Words.

tokenize

public Collection tokenize(Reader textReader)

Returns text as a Collection of Words

tokenize

public Collection tokenize(InputStream textInputStream)

Returns text as a Collection of Words