org.apache.spark.ml.feature

Class CountVectorizer

java.lang.Object

org.apache.spark.ml.PipelineStage

org.apache.spark.ml.Estimator<CountVectorizerModel>

org.apache.spark.ml.feature.CountVectorizer

All Implemented Interfaces:

java.io.Serializable, Logging, Params, Identifiable

public class CountVectorizer
extends Estimator<CountVectorizerModel>

:: Experimental :: Extracts a vocabulary from document collections and generates a CountVectorizerModel.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
CountVectorizer()
CountVectorizer(java.lang.String uid)

Method Summary

Methods

Modifier and Type	Method and Description
CountVectorizer	copy(ParamMap extra) Creates a copy of this instance with the same UID and some extra params.
CountVectorizerModel	fit(DataFrame dataset) Fits a model to the input data.
double	getMinDF()
double	getMinTF()
int	getVocabSize()
static CountVectorizer	load(java.lang.String path)
DoubleParam	minDF() Specifies the minimum number of different documents a term must appear in to be included in the vocabulary.
DoubleParam	minTF() Filter to ignore rare words in a document.
CountVectorizer	setInputCol(java.lang.String value)
CountVectorizer	setMinDF(double value)
CountVectorizer	setMinTF(double value)
CountVectorizer	setOutputCol(java.lang.String value)
CountVectorizer	setVocabSize(int value)
StructType	transformSchema(StructType schema) :: DeveloperApi ::
java.lang.String	uid() An immutable unique ID for the object and its derivatives.
StructType	validateAndTransformSchema(StructType schema) Validates and transforms the input schema.
IntParam	vocabSize() Max size of the vocabulary.

Methods inherited from class org.apache.spark.ml.Estimator

fit, fit, fit, fit

Methods inherited from class org.apache.spark.ml.PipelineStage

transformSchema

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.Params

clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn, validateParams

Methods inherited from interface org.apache.spark.ml.util.Identifiable

toString

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

CountVectorizer

public CountVectorizer(java.lang.String uid)

CountVectorizer

public CountVectorizer()

Method Detail

load

public static CountVectorizer load(java.lang.String path)

uid

public java.lang.String uid()

Description copied from interface: Identifiable

An immutable unique ID for the object and its derivatives.

Specified by:

uid in interface Identifiable

Returns:

(undocumented)

setInputCol

public CountVectorizer setInputCol(java.lang.String value)

setOutputCol

public CountVectorizer setOutputCol(java.lang.String value)

setVocabSize

public CountVectorizer setVocabSize(int value)

setMinDF

public CountVectorizer setMinDF(double value)

setMinTF

public CountVectorizer setMinTF(double value)

fit

public CountVectorizerModel fit(DataFrame dataset)

Description copied from class: Estimator

Fits a model to the input data.

Specified by:

fit in class Estimator<CountVectorizerModel>

Parameters:

dataset - (undocumented)

Returns:

(undocumented)

transformSchema

public StructType transformSchema(StructType schema)

Description copied from class: PipelineStage

:: DeveloperApi ::

Derives the output schema from the input schema.

Specified by:

transformSchema in class PipelineStage

Parameters:

schema - (undocumented)

Returns:

(undocumented)

copy

public CountVectorizer copy(ParamMap extra)

Description copied from interface: Params

Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly.

Specified by:

copy in interface Params

Specified by:

copy in class Estimator<CountVectorizerModel>

Parameters:

extra - (undocumented)

Returns:

(undocumented)

See Also:

defaultCopy()

vocabSize

public IntParam vocabSize()

Max size of the vocabulary. CountVectorizer will build a vocabulary that only considers the top vocabSize terms ordered by term frequency across the corpus.

Default: 2^18^

Returns:

(undocumented)

getVocabSize

public int getVocabSize()

minDF

public DoubleParam minDF()

Specifies the minimum number of different documents a term must appear in to be included in the vocabulary. If this is an integer >= 1, this specifies the number of documents the term must appear in; if this is a double in [0,1), then this specifies the fraction of documents.

Default: 1

Returns:

(undocumented)

getMinDF

public double getMinDF()

validateAndTransformSchema

public StructType validateAndTransformSchema(StructType schema)

Validates and transforms the input schema.

minTF

public DoubleParam minTF()

Filter to ignore rare words in a document. For each document, terms with frequency/count less than the given threshold are ignored. If this is an integer >= 1, then this specifies a count (of times the term must appear in the document); if this is a double in [0,1), then this specifies a fraction (out of the document's token count).

Note that the parameter is only used in transform of CountVectorizerModel and does not affect fitting.

Default: 1

Returns:

(undocumented)

getMinTF

public double getMinTF()