LocalLDAModel (Spark 2.3.4 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Transformer
  - - org.apache.spark.ml.Model<LDAModel>
    - - org.apache.spark.ml.clustering.LDAModel
      - org.apache.spark.ml.clustering.LocalLDAModel

All Implemented Interfaces:

java.io.Serializable, Logging, Params, Identifiable, MLWritable
```
public class LocalLDAModel
extends LDAModel
```
Local (non-distributed) model fitted by LDA.
This model stores the inferred topics only; it does not store info about the training dataset.

See Also:

Serialized Form

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static IntParam`	`checkpointInterval()`
`static Params`	`clear(Param<?> param)`
`LocalLDAModel`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static Dataset<Row>`	`describeTopics()`
`static Dataset<Row>`	`describeTopics(int maxTermsPerTopic)`
`static DoubleArrayParam`	`docConcentration()`
`DoubleArrayParam`	`docConcentration()` Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
`static Vector`	`estimatedDocConcentration()`
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`static Param<String>`	`featuresCol()`
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static int`	`getCheckpointInterval()`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static double[]`	`getDocConcentration()`
`double[]`	`getDocConcentration()`
`static String`	`getFeaturesCol()`
`static int`	`getK()`
`int`	`getK()`
`static boolean`	`getKeepLastCheckpoint()`
`boolean`	`getKeepLastCheckpoint()`
`static double`	`getLearningDecay()`
`double`	`getLearningDecay()`
`static double`	`getLearningOffset()`
`double`	`getLearningOffset()`
`static int`	`getMaxIter()`
`Vector`	`getOldDocConcentration()` Get docConcentration used by spark.mllib LDA
`LDAOptimizer`	`getOldOptimizer()`
`double`	`getOldTopicConcentration()` Get topicConcentration used by spark.mllib LDA
`static boolean`	`getOptimizeDocConcentration()`
`boolean`	`getOptimizeDocConcentration()`
`static String`	`getOptimizer()`
`String`	`getOptimizer()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static Param<Object>`	`getParam(String paramName)`
`static long`	`getSeed()`
`static double`	`getSubsamplingRate()`
`double`	`getSubsamplingRate()`
`static double`	`getTopicConcentration()`
`double`	`getTopicConcentration()`
`static String`	`getTopicDistributionCol()`
`String`	`getTopicDistributionCol()`
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static boolean`	`hasParent()`
`static boolean`	`isDefined(Param<?> param)`
`boolean`	`isDistributed()` Indicates whether this instance is of type `DistributedLDAModel`
`static boolean`	`isSet(Param<?> param)`
`static IntParam`	`k()`
`IntParam`	`k()` Param for the number of topics (clusters) to infer.
`static BooleanParam`	`keepLastCheckpoint()`
`BooleanParam`	`keepLastCheckpoint()` For EM optimizer only: `optimizer` = "em".
`static DoubleParam`	`learningDecay()`
`DoubleParam`	`learningDecay()` For Online optimizer only: `optimizer` = "online".
`static DoubleParam`	`learningOffset()`
`DoubleParam`	`learningOffset()` For Online optimizer only: `optimizer` = "online".
`static LocalLDAModel`	`load(String path)`
`static double`	`logLikelihood(Dataset<?> dataset)`
`static double`	`logPerplexity(Dataset<?> dataset)`
`static IntParam`	`maxIter()`
`static BooleanParam`	`optimizeDocConcentration()`
`BooleanParam`	`optimizeDocConcentration()` For Online optimizer only (currently): `optimizer` = "online".
`static Param<String>`	`optimizer()`
`Param<String>`	`optimizer()` Optimizer or inference algorithm used to estimate the LDA model.
`static Param<?>[]`	`params()`
`static void`	`parent_$eq(Estimator<M> x$1)`
`static Estimator<M>`	`parent()`
`static MLReader<LocalLDAModel>`	`read()`
`static void`	`save(String path)`
`static LongParam`	`seed()`
`static <T> Params`	`set(Param<T> param, T value)`
`static LDAModel`	`setFeaturesCol(String value)`
`static M`	`setParent(Estimator<M> parent)`
`static LDAModel`	`setSeed(long value)`
`static LDAModel`	`setTopicDistributionCol(String value)`
`static DoubleParam`	`subsamplingRate()`
`DoubleParam`	`subsamplingRate()` For Online optimizer only: `optimizer` = "online".
`static String[]`	`supportedOptimizers()`
`String[]`	`supportedOptimizers()` Supported values for Param `optimizer`.
`static DoubleParam`	`topicConcentration()`
`DoubleParam`	`topicConcentration()` Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
`static Param<String>`	`topicDistributionCol()`
`Param<String>`	`topicDistributionCol()` Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature).
`static Matrix`	`topicsMatrix()`
`static String`	`toString()`
`static Dataset<Row>`	`transform(Dataset<?> dataset)`
`static Dataset<Row>`	`transform(Dataset<?> dataset, ParamMap paramMap)`
`static Dataset<Row>`	`transform(Dataset<?> dataset, ParamPair<?> firstParamPair, ParamPair<?>... otherParamPairs)`
`static Dataset<Row>`	`transform(Dataset<?> dataset, ParamPair<?> firstParamPair, scala.collection.Seq<ParamPair<?>> otherParamPairs)`
`static StructType`	`transformSchema(StructType schema)`
`static String`	`uid()`
`StructType`	`validateAndTransformSchema(StructType schema)` Validates and transforms the input schema.
`static int`	`vocabSize()`
`MLWriter`	`write()` Returns an `MLWriter` instance for this ML instance.

Methods inherited from class org.apache.spark.ml.clustering.LDAModel
describeTopics, describeTopics, estimatedDocConcentration, logLikelihood, logPerplexity, setFeaturesCol, setSeed, setTopicDistributionCol, topicsMatrix, transform, transformSchema, uid, vocabSize

Methods inherited from class org.apache.spark.ml.Model
hasParent, parent, setParent

Methods inherited from class org.apache.spark.ml.Transformer
transform, transform, transform

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol
featuresCol, getFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter
getMaxIter, maxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed
getSeed, seed

Methods inherited from interface org.apache.spark.ml.param.shared.HasCheckpointInterval
checkpointInterval, getCheckpointInterval

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

- Method Detail
  - read
```
public static MLReader<LocalLDAModel> read()
```
  - load
```
public static LocalLDAModel load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - transform
```
public static Dataset<Row> transform(Dataset<?> dataset,
                                     ParamPair<?> firstParamPair,
                                     scala.collection.Seq<ParamPair<?>> otherParamPairs)
```
  - transform
```
public static Dataset<Row> transform(Dataset<?> dataset,
                                     ParamMap paramMap)
```
  - transform
```
public static Dataset<Row> transform(Dataset<?> dataset,
                                     ParamPair<?> firstParamPair,
                                     ParamPair<?>... otherParamPairs)
```
  - parent
```
public static Estimator<M> parent()
```
  - parent_$eq
```
public static void parent_$eq(Estimator<M> x$1)
```
  - setParent
```
public static M setParent(Estimator<M> parent)
```
  - hasParent
```
public static boolean hasParent()
```
  - featuresCol
```
public static final Param<String> featuresCol()
```
  - getFeaturesCol
```
public static final String getFeaturesCol()
```
  - maxIter
```
public static final IntParam maxIter()
```
  - getMaxIter
```
public static final int getMaxIter()
```
  - seed
```
public static final LongParam seed()
```
  - getSeed
```
public static final long getSeed()
```
  - checkpointInterval
```
public static final IntParam checkpointInterval()
```
  - getCheckpointInterval
```
public static final int getCheckpointInterval()
```
  - k
```
public static final IntParam k()
```
  - getK
```
public static int getK()
```
  - docConcentration
```
public static final DoubleArrayParam docConcentration()
```
  - getDocConcentration
```
public static double[] getDocConcentration()
```
  - topicConcentration
```
public static final DoubleParam topicConcentration()
```
  - getTopicConcentration
```
public static double getTopicConcentration()
```
  - supportedOptimizers
```
public static final String[] supportedOptimizers()
```
  - optimizer
```
public static final Param<String> optimizer()
```
  - getOptimizer
```
public static String getOptimizer()
```
  - topicDistributionCol
```
public static final Param<String> topicDistributionCol()
```
  - getTopicDistributionCol
```
public static String getTopicDistributionCol()
```
  - learningOffset
```
public static final DoubleParam learningOffset()
```
  - getLearningOffset
```
public static double getLearningOffset()
```
  - learningDecay
```
public static final DoubleParam learningDecay()
```
  - getLearningDecay
```
public static double getLearningDecay()
```
  - subsamplingRate
```
public static final DoubleParam subsamplingRate()
```
  - getSubsamplingRate
```
public static double getSubsamplingRate()
```
  - optimizeDocConcentration
```
public static final BooleanParam optimizeDocConcentration()
```
  - getOptimizeDocConcentration
```
public static boolean getOptimizeDocConcentration()
```
  - keepLastCheckpoint
```
public static final BooleanParam keepLastCheckpoint()
```
  - getKeepLastCheckpoint
```
public static boolean getKeepLastCheckpoint()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - uid
```
public static String uid()
```
  - vocabSize
```
public static int vocabSize()
```
  - setFeaturesCol
```
public static LDAModel setFeaturesCol(String value)
```
  - setTopicDistributionCol
```
public static LDAModel setTopicDistributionCol(String value)
```
  - setSeed
```
public static LDAModel setSeed(long value)
```
  - transform
```
public static Dataset<Row> transform(Dataset<?> dataset)
```
  - transformSchema
```
public static StructType transformSchema(StructType schema)
```
  - estimatedDocConcentration
```
public static Vector estimatedDocConcentration()
```
  - topicsMatrix
```
public static Matrix topicsMatrix()
```
  - logLikelihood
```
public static double logLikelihood(Dataset<?> dataset)
```
  - logPerplexity
```
public static double logPerplexity(Dataset<?> dataset)
```
  - describeTopics
```
public static Dataset<Row> describeTopics(int maxTermsPerTopic)
```
  - describeTopics
```
public static Dataset<Row> describeTopics()
```
  - copy
```
public LocalLDAModel copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Model<LDAModel>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - isDistributed
```
public boolean isDistributed()
```
    Description copied from class: LDAModel
    
    Indicates whether this instance is of type DistributedLDAModel
    
    Specified by:
    
    isDistributed in class LDAModel
  - write
```
public MLWriter write()
```
    Description copied from interface: MLWritable
    
    Returns an MLWriter instance for this ML instance.
    
    Returns:
    
    (undocumented)
  - k
```
public IntParam k()
```
    Param for the number of topics (clusters) to infer. Must be > 1. Default: 10.
    
    Returns:
    
    (undocumented)
  - getK
```
public int getK()
```
  - docConcentration
```
public DoubleArrayParam docConcentration()
```
    Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
    This is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
    If not set by the user, then docConcentration is set automatically. If set to singleton vector [alpha], then alpha is replicated to a vector of length k in fitting. Otherwise, the docConcentration vector must be length k. (default = automatic)
    Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.
    
    Returns:
    
    (undocumented)
  - getDocConcentration
```
public double[] getDocConcentration()
```
  - getOldDocConcentration
```
public Vector getOldDocConcentration()
```
    Get docConcentration used by spark.mllib LDA
  - topicConcentration
```
public DoubleParam topicConcentration()
```
    Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
    This is the parameter to a symmetric Dirichlet distribution.
    Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
    If not set by the user, then topicConcentration is set automatically. (default = automatic)
    Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.
    
    Returns:
    
    (undocumented)
  - getTopicConcentration
```
public double getTopicConcentration()
```
  - getOldTopicConcentration
```
public double getOldTopicConcentration()
```
    Get topicConcentration used by spark.mllib LDA
  - supportedOptimizers
```
public String[] supportedOptimizers()
```
    Supported values for Param optimizer.
  - optimizer
```
public Param<String> optimizer()
```
    Optimizer or inference algorithm used to estimate the LDA model. Currently supported (case-insensitive): - "online": Online Variational Bayes (default) - "em": Expectation-Maximization
    For details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here
    
    Returns:
    
    (undocumented)
  - getOptimizer
```
public String getOptimizer()
```
  - topicDistributionCol
```
public Param<String> topicDistributionCol()
```
    Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.
    This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
    
    Returns:
    
    (undocumented)
  - getTopicDistributionCol
```
public String getTopicDistributionCol()
```
  - learningOffset
```
public DoubleParam learningOffset()
```
    For Online optimizer only: optimizer = "online".
    A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.
    
    Returns:
    
    (undocumented)
  - getLearningOffset
```
public double getLearningOffset()
```
  - learningDecay
```
public DoubleParam learningDecay()
```
    For Online optimizer only: optimizer = "online".
    Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.
    
    Returns:
    
    (undocumented)
  - getLearningDecay
```
public double getLearningDecay()
```
  - subsamplingRate
```
public DoubleParam subsamplingRate()
```
    For Online optimizer only: optimizer = "online".
    Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].
    Note that this should be adjusted in synch with LDA.maxIter so the entire corpus is used. Specifically, set both so that maxIterations * miniBatchFraction greater than or equal to 1.
    Note: This is the same as the miniBatchFraction parameter in OnlineLDAOptimizer.
    Default: 0.05, i.e., 5% of total documents.
    
    Returns:
    
    (undocumented)
  - getSubsamplingRate
```
public double getSubsamplingRate()
```
  - optimizeDocConcentration
```
public BooleanParam optimizeDocConcentration()
```
    For Online optimizer only (currently): optimizer = "online".
    Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false
    
    Returns:
    
    (undocumented)
  - getOptimizeDocConcentration
```
public boolean getOptimizeDocConcentration()
```
  - keepLastCheckpoint
```
public BooleanParam keepLastCheckpoint()
```
    For EM optimizer only: optimizer = "em".
    If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.
    See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.
    Default: true
    
    Returns:
    
    (undocumented)
  - getKeepLastCheckpoint
```
public boolean getKeepLastCheckpoint()
```
  - validateAndTransformSchema
```
public StructType validateAndTransformSchema(StructType schema)
```
    Validates and transforms the input schema.
    
    Parameters:
    
    schema - input schema
    
    Returns:
    
    output schema
  - getOldOptimizer
```
public LDAOptimizer getOldOptimizer()
```

Class LocalLDAModel

Method Summary

Methods inherited from class org.apache.spark.ml.clustering.LDAModel

Methods inherited from class org.apache.spark.ml.Model

Methods inherited from class org.apache.spark.ml.Transformer

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasFeaturesCol

Methods inherited from interface org.apache.spark.ml.param.shared.HasMaxIter

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed

Methods inherited from interface org.apache.spark.ml.param.shared.HasCheckpointInterval

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.internal.Logging

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Method Detail

read

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

transform

transform

transform

parent

parent_$eq

setParent

hasParent

featuresCol

getFeaturesCol

maxIter

getMaxIter

seed

getSeed

checkpointInterval

getCheckpointInterval

k

getK

docConcentration

getDocConcentration

topicConcentration

getTopicConcentration

supportedOptimizers

optimizer

getOptimizer

topicDistributionCol

getTopicDistributionCol

learningOffset

getLearningOffset

learningDecay

getLearningDecay

subsamplingRate

getSubsamplingRate

optimizeDocConcentration

getOptimizeDocConcentration

keepLastCheckpoint

getKeepLastCheckpoint

save

uid

vocabSize

setFeaturesCol

setTopicDistributionCol

setSeed

transform

transformSchema

estimatedDocConcentration

topicsMatrix

logLikelihood