VectorIndexer (Spark 2.3.4 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<VectorIndexerModel>
  - - org.apache.spark.ml.feature.VectorIndexer

All Implemented Interfaces:

java.io.Serializable, Logging, Params, DefaultParamsWritable, Identifiable, MLWritable
```
public class VectorIndexer
extends Estimator<VectorIndexerModel>
implements DefaultParamsWritable
```
Class for indexing categorical feature columns in a dataset of Vector.
This has 2 usage modes: - Automatically identify categorical features (default behavior) - This helps process a dataset of unknown vectors into a dataset with some continuous features and some categorical features. The choice between continuous and categorical is based upon a maxCategories parameter. - Set maxCategories to the maximum number of categorical any categorical feature should have. - E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}. If maxCategories = 2, then feature 0 will be declared categorical and use indices {0, 1}, and feature 1 will be declared continuous. - Index all features, if all features are categorical - If maxCategories is set to be very large, then this will build an index of unique values for all features. - Warning: This can cause problems if features are continuous since this will collect ALL unique values to the driver. - E.g.: Feature 0 has unique values {-1.0, 0.0}, and feature 1 values {1.0, 3.0, 5.0}. If maxCategories is greater than or equal to 3, then both features will be declared categorical.
This returns a model which can transform categorical features to use 0-based indices.
Index stability: - This is not guaranteed to choose the same category index across multiple runs. - If a categorical feature includes value 0, then this is guaranteed to map value 0 to index 0. This maintains vector sparsity. - More stability may be added in the future.
TODO: Future extensions: The following functionality is planned for the future: - Preserve metadata in transform; if a feature's metadata is already present, do not recompute. - Specify certain features to not index, either via a parameter or via existing metadata. - Add warning if a categorical feature has only 1 category.

See Also:

Serialized Form

Constructor Summary

Constructors
Constructor and Description

VectorIndexer()

VectorIndexer(String uid)

Constructors
Constructor and Description
`VectorIndexer()`
`VectorIndexer(String uid)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`VectorIndexer`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`VectorIndexerModel`	`fit(Dataset<?> dataset)` Fits a model to the input data.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getHandleInvalid()`
`static String`	`getInputCol()`
`static int`	`getMaxCategories()`
`int`	`getMaxCategories()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static String`	`getOutputCol()`
`static Param<Object>`	`getParam(String paramName)`
`static Param<String>`	`handleInvalid()`
`Param<String>`	`handleInvalid()` Param for how to handle invalid data (unseen labels or NULL values).
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static Param<String>`	`inputCol()`
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static VectorIndexer`	`load(String path)`
`static IntParam`	`maxCategories()`
`IntParam`	`maxCategories()` Threshold for the number of values a categorical feature can take.
`static Param<String>`	`outputCol()`
`static Param<?>[]`	`params()`
`static void`	`save(String path)`
`static <T> Params`	`set(Param<T> param, T value)`
`VectorIndexer`	`setHandleInvalid(String value)`
`VectorIndexer`	`setInputCol(String value)`
`VectorIndexer`	`setMaxCategories(int value)`
`VectorIndexer`	`setOutputCol(String value)`
`static String`	`toString()`
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`static MLWriter`	`write()`

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol
getInputCol, inputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol
getOutputCol, outputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasHandleInvalid
getHandleInvalid

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable
write

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Constructor Detail
  - VectorIndexer
```
public VectorIndexer(String uid)
```
  - VectorIndexer
```
public VectorIndexer()
```
- Method Detail
  - load
```
public static VectorIndexer load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - inputCol
```
public static final Param<String> inputCol()
```
  - getInputCol
```
public static final String getInputCol()
```
  - outputCol
```
public static final Param<String> outputCol()
```
  - getOutputCol
```
public static final String getOutputCol()
```
  - getHandleInvalid
```
public static final String getHandleInvalid()
```
  - handleInvalid
```
public static Param<String> handleInvalid()
```
  - maxCategories
```
public static IntParam maxCategories()
```
  - getMaxCategories
```
public static int getMaxCategories()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - write
```
public static MLWriter write()
```
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    
    (undocumented)
  - setMaxCategories
```
public VectorIndexer setMaxCategories(int value)
```
  - setInputCol
```
public VectorIndexer setInputCol(String value)
```
  - setOutputCol
```
public VectorIndexer setOutputCol(String value)
```
  - setHandleInvalid
```
public VectorIndexer setHandleInvalid(String value)
```
  - fit
```
public VectorIndexerModel fit(Dataset<?> dataset)
```
    Description copied from class: Estimator
    
    Fits a model to the input data.
    
    Specified by:
    
    fit in class Estimator<VectorIndexerModel>
    
    Parameters:
    
    dataset - (undocumented)
    
    Returns:
    
    (undocumented)
  - transformSchema
```
public StructType transformSchema(StructType schema)
```
    Description copied from class: PipelineStage
    
    :: DeveloperApi ::
    Check transform validity and derive the output schema from the input schema.
    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
    
    Specified by:
    
    transformSchema in class PipelineStage
    
    Parameters:
    
    schema - (undocumented)
    
    Returns:
    
    (undocumented)
  - copy
```
public VectorIndexer copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Estimator<VectorIndexerModel>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - handleInvalid
```
public Param<String> handleInvalid()
```
    Param for how to handle invalid data (unseen labels or NULL values). Note: this param only applies to categorical features, not continuous ones. Options are: 'skip': filter out rows with invalid data. 'error': throw an error. 'keep': put invalid data in a special additional bucket, at index of the number of categories of the feature. Default value: "error"
    
    Specified by:
    
    handleInvalid in interface HasHandleInvalid
    
    Returns:
    
    (undocumented)
  - maxCategories
```
public IntParam maxCategories()
```
    Threshold for the number of values a categorical feature can take. If a feature is found to have > maxCategories values, then it is declared continuous. Must be greater than or equal to 2.
    (default = 20)
    
    Returns:
    
    (undocumented)
  - getMaxCategories
```
public int getMaxCategories()
```

Class VectorIndexer

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasHandleInvalid

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.util.DefaultParamsWritable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.internal.Logging

Constructor Detail

VectorIndexer

VectorIndexer

Method Detail

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

inputCol

getInputCol

outputCol

getOutputCol

getHandleInvalid

handleInvalid

maxCategories

getMaxCategories

save

write

uid

setMaxCategories

setInputCol

setOutputCol

setHandleInvalid

fit

transformSchema

copy

handleInvalid

maxCategories

getMaxCategories