Word2VecModel (Spark 2.3.3 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Transformer
  - - org.apache.spark.ml.Model<Word2VecModel>
    - - org.apache.spark.ml.feature.Word2VecModel

All Implemented Interfaces:

java.io.Serializable, Logging, Params, Identifiable, MLWritable
```
public class Word2VecModel
extends Model<Word2VecModel>
implements MLWritable
```
Model fitted by Word2Vec.

See Also:

Serialized Form

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class Word2VecModel.Word2VecModelWriter$

Nested Classes
Modifier and Type	Class and Description
`static class`	`Word2VecModel.Word2VecModelWriter$`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`Word2VecModel`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`Dataset<Row>`	`findSynonyms(String word, int num)` Find "num" number of words closest in similarity to the given word, not including the word itself.
`Dataset<Row>`	`findSynonyms(Vector vec, int num)` Find "num" number of words whose vector representation is most similar to the supplied vector.
`scala.Tuple2<String,Object>[]`	`findSynonymsArray(String word, int num)` Find "num" number of words closest in similarity to the given word, not including the word itself.
`scala.Tuple2<String,Object>[]`	`findSynonymsArray(Vector vec, int num)` Find "num" number of words whose vector representation is most similar to the supplied vector.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static String`	`getInputCol()`
`static int`	`getMaxIter()`
`static int`	`getMaxSentenceLength()`
`int`	`getMaxSentenceLength()`
`static int`	`getMinCount()`
`int`	`getMinCount()`
`static int`	`getNumPartitions()`
`int`	`getNumPartitions()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static String`	`getOutputCol()`
`static Param<Object>`	`getParam(String paramName)`
`static long`	`getSeed()`
`static double`	`getStepSize()`
`Dataset<Row>`	`getVectors()` Returns a dataframe with two fields, "word" and "vector", with "word" being a String and and the vector the DenseVector that it is mapped to.
`static int`	`getVectorSize()`
`int`	`getVectorSize()`
`static int`	`getWindowSize()`
`int`	`getWindowSize()`
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static boolean`	`hasParent()`
`static Param<String>`	`inputCol()`
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static Word2VecModel`	`load(String path)`
`static IntParam`	`maxIter()`
`static IntParam`	`maxSentenceLength()`
`IntParam`	`maxSentenceLength()` Sets the maximum length (in words) of each sentence in the input data.
`static IntParam`	`minCount()`
`IntParam`	`minCount()` The minimum number of times a token must appear to be included in the word2vec model's vocabulary.
`static IntParam`	`numPartitions()`
`IntParam`	`numPartitions()` Number of partitions for sentences of words.
`static Param<String>`	`outputCol()`
`static Param<?>[]`	`params()`
`static void`	`parent_$eq(Estimator<M> x$1)`
`static Estimator<M>`	`parent()`
`static MLReader<Word2VecModel>`	`read()`
`static void`	`save(String path)`
`static LongParam`	`seed()`
`static <T> Params`	`set(Param<T> param, T value)`
`Word2VecModel`	`setInputCol(String value)`
`Word2VecModel`	`setOutputCol(String value)`
`static M`	`setParent(Estimator<M> parent)`
`static DoubleParam`	`stepSize()`
`static String`	`toString()`
`Dataset<Row>`	`transform(Dataset<?> dataset)` Transform a sentence column to a vector column to represent the whole sentence.
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`StructType`	`validateAndTransformSchema(StructType schema)` Validate and transform the input schema.
`static IntParam`	`vectorSize()`
`IntParam`	`vectorSize()` The dimension of the code that you want to transform from words.
`static IntParam`	`windowSize()`
`IntParam`	`windowSize()` The window size (context words from [-window, window]).
`MLWriter`	`write()` Returns an `MLWriter` instance for this ML instance.

Methods inherited from class org.apache.spark.ml.Model
hasParent, parent, setParent

Methods inherited from class org.apache.spark.ml.Transformer
transform, transform, transform

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol
getInputCol, inputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol
getOutputCol, outputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter
getMaxIter, maxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasStepSize
getStepSize, stepSize

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed
getSeed, seed

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Method Detail
  - read
```
public static MLReader<Word2VecModel> read()
```
  - load
```
public static Word2VecModel load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - parent
```
public static Estimator<M> parent()
```
  - parent_$eq
```
public static void parent_$eq(Estimator<M> x$1)
```
  - setParent
```
public static M setParent(Estimator<M> parent)
```
  - hasParent
```
public static boolean hasParent()
```
  - inputCol
```
public static final Param<String> inputCol()
```
  - getInputCol
```
public static final String getInputCol()
```
  - outputCol
```
public static final Param<String> outputCol()
```
  - getOutputCol
```
public static final String getOutputCol()
```
  - maxIter
```
public static final IntParam maxIter()
```
  - getMaxIter
```
public static final int getMaxIter()
```
  - stepSize
```
public static DoubleParam stepSize()
```
  - getStepSize
```
public static final double getStepSize()
```
  - seed
```
public static final LongParam seed()
```
  - getSeed
```
public static final long getSeed()
```
  - vectorSize
```
public static final IntParam vectorSize()
```
  - getVectorSize
```
public static int getVectorSize()
```
  - windowSize
```
public static final IntParam windowSize()
```
  - getWindowSize
```
public static int getWindowSize()
```
  - numPartitions
```
public static final IntParam numPartitions()
```
  - getNumPartitions
```
public static int getNumPartitions()
```
  - minCount
```
public static final IntParam minCount()
```
  - getMinCount
```
public static int getMinCount()
```
  - maxSentenceLength
```
public static final IntParam maxSentenceLength()
```
  - getMaxSentenceLength
```
public static int getMaxSentenceLength()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    
    (undocumented)
  - getVectors
```
public Dataset<Row> getVectors()
```
    Returns a dataframe with two fields, "word" and "vector", with "word" being a String and and the vector the DenseVector that it is mapped to.
    
    Returns:
    
    (undocumented)
  - findSynonyms
```
public Dataset<Row> findSynonyms(String word,
                                 int num)
```
    Find "num" number of words closest in similarity to the given word, not including the word itself.
    
    Parameters:
    
    word - (undocumented)
    
    num - (undocumented)
    
    Returns:
    
    a dataframe with columns "word" and "similarity" of the word and the cosine similarities between the synonyms and the given word.
  - findSynonyms
```
public Dataset<Row> findSynonyms(Vector vec,
                                 int num)
```
    Find "num" number of words whose vector representation is most similar to the supplied vector. If the supplied vector is the vector representation of a word in the model's vocabulary, that word will be in the results.
    
    Parameters:
    
    vec - (undocumented)
    
    num - (undocumented)
    
    Returns:
    
    a dataframe with columns "word" and "similarity" of the word and the cosine similarities between the synonyms and the given word vector.
  - findSynonymsArray
```
public scala.Tuple2<String,Object>[] findSynonymsArray(Vector vec,
                                                       int num)
```
    Find "num" number of words whose vector representation is most similar to the supplied vector. If the supplied vector is the vector representation of a word in the model's vocabulary, that word will be in the results.
    
    Parameters:
    
    vec - (undocumented)
    
    num - (undocumented)
    
    Returns:
    
    an array of the words and the cosine similarities between the synonyms given word vector.
  - findSynonymsArray
```
public scala.Tuple2<String,Object>[] findSynonymsArray(String word,
                                                       int num)
```
    Find "num" number of words closest in similarity to the given word, not including the word itself.
    
    Parameters:
    
    word - (undocumented)
    
    num - (undocumented)
    
    Returns:
    
    an array of the words and the cosine similarities between the synonyms given word vector.
  - setInputCol
```
public Word2VecModel setInputCol(String value)
```
  - setOutputCol
```
public Word2VecModel setOutputCol(String value)
```
  - transform
```
public Dataset<Row> transform(Dataset<?> dataset)
```
    Transform a sentence column to a vector column to represent the whole sentence. The transform is performed by averaging all word vectors it contains.
    
    Specified by:
    
    transform in class Transformer
    
    Parameters:
    
    dataset - (undocumented)
    
    Returns:
    
    (undocumented)
  - transformSchema
```
public StructType transformSchema(StructType schema)
```
    Description copied from class: PipelineStage
    
    :: DeveloperApi ::
    Check transform validity and derive the output schema from the input schema.
    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
    
    Specified by:
    
    transformSchema in class PipelineStage
    
    Parameters:
    
    schema - (undocumented)
    
    Returns:
    
    (undocumented)
  - copy
```
public Word2VecModel copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Model<Word2VecModel>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - write
```
public MLWriter write()
```
    Description copied from interface: MLWritable
    
    Returns an MLWriter instance for this ML instance.
    
    Specified by:
    
    write in interface MLWritable
    
    Returns:
    
    (undocumented)
  - vectorSize
```
public IntParam vectorSize()
```
    The dimension of the code that you want to transform from words. Default: 100
    
    Returns:
    
    (undocumented)
  - getVectorSize
```
public int getVectorSize()
```
  - windowSize
```
public IntParam windowSize()
```
    The window size (context words from [-window, window]). Default: 5
    
    Returns:
    
    (undocumented)
  - getWindowSize
```
public int getWindowSize()
```
  - numPartitions
```
public IntParam numPartitions()
```
    Number of partitions for sentences of words. Default: 1
    
    Returns:
    
    (undocumented)
  - getNumPartitions
```
public int getNumPartitions()
```
  - minCount
```
public IntParam minCount()
```
    The minimum number of times a token must appear to be included in the word2vec model's vocabulary. Default: 5
    
    Returns:
    
    (undocumented)
  - getMinCount
```
public int getMinCount()
```
  - maxSentenceLength
```
public IntParam maxSentenceLength()
```
    Sets the maximum length (in words) of each sentence in the input data. Any sentence longer than this threshold will be divided into chunks of up to maxSentenceLength size. Default: 1000
    
    Returns:
    
    (undocumented)
  - getMaxSentenceLength
```
public int getMaxSentenceLength()
```
  - validateAndTransformSchema
```
public StructType validateAndTransformSchema(StructType schema)
```
    Validate and transform the input schema.
    
    Parameters:
    
    schema - (undocumented)
    
    Returns:
    
    (undocumented)

Class Word2VecModel

Nested Class Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Model

Methods inherited from class org.apache.spark.ml.Transformer

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasInputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasOutputCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasStepSize

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.internal.Logging

Method Detail

read

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

parent

parent_$eq

setParent

hasParent

inputCol

getInputCol

outputCol

getOutputCol

maxIter

getMaxIter

stepSize

getStepSize

seed

getSeed

vectorSize

getVectorSize

windowSize

getWindowSize

numPartitions

getNumPartitions

minCount

getMinCount

maxSentenceLength

getMaxSentenceLength

save

uid

getVectors

findSynonyms

findSynonyms

findSynonymsArray

findSynonymsArray

setInputCol

setOutputCol

transform

transformSchema

copy

write

vectorSize

getVectorSize

windowSize

getWindowSize

numPartitions

getNumPartitions

minCount

getMinCount

maxSentenceLength