pl.edu.agh.cast.importer.base.tokenizer.csv
Class CsvTokenizer

java.lang.Object
  pl.edu.agh.cast.importer.base.tokenizer.AbstractImportTokenizer
      pl.edu.agh.cast.importer.base.tokenizer.csv.CsvTokenizer

All Implemented Interfaces:

IImportTokenizer

public class CsvTokenizer
extends AbstractImportTokenizer

Simple CSV data tokenizer.

Author:

AGH CAST Team

Field Summary

static String	COMMENT_CHAR_OPTION_NAME The comment character option indicating that a line is a comment and is not supposed to be imported.
static String	CSV_TOKENIZER_ID Name of the CSV tokenizer.
static String	QUALIFIER_OPTION_NAME The text qualifier option name.
static String	RECORD_SEPARATOR_OPTION_NAME The record separator option name identifier of the CSV data importer.

Fields inherited from class pl.edu.agh.cast.importer.base.tokenizer.AbstractImportTokenizer

options

Constructor Summary

CsvTokenizer()

Method Summary

List<RawTabularData>

tokenize(InputStream dataIs, long rowsLimit, org.eclipse.core.runtime.IProgressMonitor monitor) Splits a given input stream into tokens, using specified tokenizer options.

Methods inherited from class pl.edu.agh.cast.importer.base.tokenizer.AbstractImportTokenizer

equals, getEncoding, getOptionValue, getTokenizerOptions, hashCode, removeEmptyCellsFromRowEnd, removeQualifier, setEncoding, setTokenizerOptions

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, toString, wait, wait, wait

Field Detail

CSV_TOKENIZER_ID

public static final String CSV_TOKENIZER_ID

Name of the CSV tokenizer.

See Also:

Constant Field Values

RECORD_SEPARATOR_OPTION_NAME

public static final String RECORD_SEPARATOR_OPTION_NAME

The record separator option name identifier of the CSV data importer.

See Also:

Constant Field Values

COMMENT_CHAR_OPTION_NAME

public static final String COMMENT_CHAR_OPTION_NAME

The comment character option indicating that a line is a comment and is not supposed to be imported.

See Also:

Constant Field Values

QUALIFIER_OPTION_NAME

public static final String QUALIFIER_OPTION_NAME

The text qualifier option name.

See Also:

Constant Field Values

Constructor Detail

CsvTokenizer

public CsvTokenizer()

Method Detail

tokenize

public List<RawTabularData> tokenize(InputStream dataIs,
                                     long rowsLimit,
                                     org.eclipse.core.runtime.IProgressMonitor monitor)
                              throws IOException

Splits a given input stream into tokens, using specified tokenizer options.

Parameters:

dataIs - the data input stream to tokenize

rowsLimit - the maximum number of rows to be imported

monitor - the progress monitor for the tokenization operation

Returns:

the tokenized data in an unanalyzed tabular form

Throws:

IOException

See Also:

IImportTokenizer.tokenize(java.io.InputStream, long, org.eclipse.core.runtime.IProgressMonitor)