public class LDA extends Estimator<LDAModel> implements LDAParams, DefaultParamsWritable
Terminology: - "term" = "word": an element of the vocabulary - "token": instance of a term appearing in a document - "topic": multinomial distribution over terms representing some concept - "document": one piece of text, corresponding to one row in the input data
Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
Input data (featuresCol):
LDA is given a collection of documents as input data, via the featuresCol parameter.
Each document is specified as a Vector of length vocabSize, where each entry is the
count for the corresponding term (word) in the document. Feature transformers such as
Tokenizer and CountVectorizer
can be useful for converting text to word count vectors.
| Modifier and Type | Method and Description |
|---|---|
IntParam |
checkpointInterval()
Param for set checkpoint interval (>= 1) or disable checkpoint (-1).
|
LDA |
copy(ParamMap extra)
Creates a copy of this instance with the same UID and some extra params.
|
DoubleArrayParam |
docConcentration()
Concentration parameter (commonly named "alpha") for the prior placed on documents'
distributions over topics ("theta").
|
Param<String> |
featuresCol()
Param for features column name.
|
LDAModel |
fit(Dataset<?> dataset)
Fits a model to the input data.
|
IntParam |
k()
Param for the number of topics (clusters) to infer.
|
BooleanParam |
keepLastCheckpoint()
For EM optimizer only:
optimizer = "em". |
DoubleParam |
learningDecay()
For Online optimizer only:
optimizer = "online". |
DoubleParam |
learningOffset()
For Online optimizer only:
optimizer = "online". |
static LDA |
load(String path) |
IntParam |
maxIter()
Param for maximum number of iterations (>= 0).
|
BooleanParam |
optimizeDocConcentration()
For Online optimizer only (currently):
optimizer = "online". |
Param<String> |
optimizer()
Optimizer or inference algorithm used to estimate the LDA model.
|
static MLReader<LDA> |
read() |
LongParam |
seed()
Param for random seed.
|
LDA |
setCheckpointInterval(int value) |
LDA |
setDocConcentration(double value) |
LDA |
setDocConcentration(double[] value) |
LDA |
setFeaturesCol(String value)
The features for LDA should be a
Vector representing the word counts in a document. |
LDA |
setK(int value) |
LDA |
setKeepLastCheckpoint(boolean value) |
LDA |
setLearningDecay(double value) |
LDA |
setLearningOffset(double value) |
LDA |
setMaxIter(int value) |
LDA |
setOptimizeDocConcentration(boolean value) |
LDA |
setOptimizer(String value) |
LDA |
setSeed(long value) |
LDA |
setSubsamplingRate(double value) |
LDA |
setTopicConcentration(double value) |
LDA |
setTopicDistributionCol(String value) |
DoubleParam |
subsamplingRate()
For Online optimizer only:
optimizer = "online". |
String[] |
supportedOptimizers()
Supported values for Param
optimizer. |
DoubleParam |
topicConcentration()
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics'
distributions over terms.
|
Param<String> |
topicDistributionCol()
Output column with estimates of the topic mixture distribution for each document (often called
"theta" in the literature).
|
StructType |
transformSchema(StructType schema)
Check transform validity and derive the output schema from the input schema.
|
String |
uid()
An immutable unique ID for the object and its derivatives.
|
paramsequals, getClass, hashCode, notify, notifyAll, toString, wait, wait, waitgetDocConcentration, getK, getKeepLastCheckpoint, getLearningDecay, getLearningOffset, getOldDocConcentration, getOldOptimizer, getOldTopicConcentration, getOptimizeDocConcentration, getOptimizer, getSubsamplingRate, getTopicConcentration, getTopicDistributionCol, validateAndTransformSchemagetFeaturesColgetMaxItergetCheckpointIntervalclear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwntoStringwritesave$init$, initializeForcefully, initializeLogIfNecessary, initializeLogIfNecessary, initializeLogIfNecessary$default$2, initLock, isTraceEnabled, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning, org$apache$spark$internal$Logging$$log__$eq, org$apache$spark$internal$Logging$$log_, uninitializepublic static LDA load(String path)
public final IntParam k()
LDAParamspublic final DoubleArrayParam docConcentration()
LDAParamsThis is the parameter to a Dirichlet distribution, where larger values mean more smoothing (more regularization).
If not set by the user, then docConcentration is set automatically. If set to
singleton vector [alpha], then alpha is replicated to a vector of length k in fitting.
Otherwise, the docConcentration vector must be length k.
(default = automatic)
Optimizer-specific parameter settings: - EM - Currently only supports symmetric distributions, so all values in the vector should be the same. - Values should be greater than 1.0 - default = uniformly (50 / k) + 1, where 50/k is common in LDA libraries and +1 follows from Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Values should be greater than or equal to 0 - default = uniformly (1.0 / k), following the implementation from here.
docConcentration in interface LDAParamspublic final DoubleParam topicConcentration()
LDAParamsThis is the parameter to a symmetric Dirichlet distribution.
Note: The topics' distributions over terms are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many later papers such as Asuncion et al., 2009.
If not set by the user, then topicConcentration is set automatically. (default = automatic)
Optimizer-specific parameter settings: - EM - Value should be greater than 1.0 - default = 0.1 + 1, where 0.1 gives a small amount of smoothing and +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM. - Online - Value should be greater than or equal to 0 - default = (1.0 / k), following the implementation from here.
topicConcentration in interface LDAParamspublic final String[] supportedOptimizers()
LDAParamsoptimizer.supportedOptimizers in interface LDAParamspublic final Param<String> optimizer()
LDAParamsFor details, see the following papers: - Online LDA: Hoffman, Blei and Bach. "Online Learning for Latent Dirichlet Allocation." Neural Information Processing Systems, 2010. See here - EM: Asuncion et al. "On Smoothing and Inference for Topic Models." Uncertainty in Artificial Intelligence, 2009. See here
public final Param<String> topicDistributionCol()
LDAParamsThis uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.
topicDistributionCol in interface LDAParamspublic final DoubleParam learningOffset()
LDAParamsoptimizer = "online".
A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less. This is called "tau0" in the Online LDA paper (Hoffman et al., 2010) Default: 1024, following Hoffman et al.
learningOffset in interface LDAParamspublic final DoubleParam learningDecay()
LDAParamsoptimizer = "online".
Learning rate, set as an exponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence. This is called "kappa" in the Online LDA paper (Hoffman et al., 2010). Default: 0.51, based on Hoffman et al.
learningDecay in interface LDAParamspublic final DoubleParam subsamplingRate()
LDAParamsoptimizer = "online".
Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].
Note that this should be adjusted in synch with LDA.maxIter
so the entire corpus is used. Specifically, set both so that
maxIterations * miniBatchFraction greater than or equal to 1.
Note: This is the same as the miniBatchFraction parameter in
OnlineLDAOptimizer.
Default: 0.05, i.e., 5% of total documents.
subsamplingRate in interface LDAParamspublic final BooleanParam optimizeDocConcentration()
LDAParamsoptimizer = "online".
Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training. Setting this to true will make the model more expressive and fit the training data better. Default: false
optimizeDocConcentration in interface LDAParamspublic final BooleanParam keepLastCheckpoint()
LDAParamsoptimizer = "em".
If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care. Note that checkpoints will be cleaned up via reference counting, regardless.
See DistributedLDAModel.getCheckpointFiles for getting remaining checkpoints and
DistributedLDAModel.deleteCheckpointFiles for removing remaining checkpoints.
Default: true
keepLastCheckpoint in interface LDAParamspublic final IntParam checkpointInterval()
HasCheckpointIntervalcheckpointInterval in interface HasCheckpointIntervalpublic final LongParam seed()
HasSeedpublic final IntParam maxIter()
HasMaxItermaxIter in interface HasMaxIterpublic final Param<String> featuresCol()
HasFeaturesColfeaturesCol in interface HasFeaturesColpublic String uid()
Identifiableuid in interface Identifiablepublic LDA setFeaturesCol(String value)
Vector representing the word counts in a document.
The vector should be of length vocabSize, with counts for each term (word).
value - (undocumented)public LDA setMaxIter(int value)
public LDA setSeed(long value)
public LDA setCheckpointInterval(int value)
public LDA setK(int value)
public LDA setDocConcentration(double[] value)
public LDA setDocConcentration(double value)
public LDA setTopicConcentration(double value)
public LDA setOptimizer(String value)
public LDA setTopicDistributionCol(String value)
public LDA setLearningOffset(double value)
public LDA setLearningDecay(double value)
public LDA setSubsamplingRate(double value)
public LDA setOptimizeDocConcentration(boolean value)
public LDA setKeepLastCheckpoint(boolean value)
public LDA copy(ParamMap extra)
ParamsdefaultCopy().public LDAModel fit(Dataset<?> dataset)
Estimatorpublic StructType transformSchema(StructType schema)
PipelineStage
We check validity for interactions between parameters during transformSchema and
raise an exception if any parameter value is invalid. Parameter value checks which
do not depend on other parameters are handled by Param.validate().
Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
transformSchema in class PipelineStageschema - (undocumented)