org.apache.spark.streaming.dstream

Class ReducedWindowedDStream<K,V>

Object

org.apache.spark.streaming.dstream.DStream<scala.Tuple2<K,V>>

org.apache.spark.streaming.dstream.ReducedWindowedDStream<K,V>

All Implemented Interfaces:

java.io.Serializable, Logging

public class ReducedWindowedDStream<K,V>
extends DStream<scala.Tuple2<K,V>>

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
ReducedWindowedDStream(DStream<scala.Tuple2<K,V>> parent, scala.Function2<V,V,V> reduceFunc, scala.Function2<V,V,V> invReduceFunc, scala.Option<scala.Function1<scala.Tuple2<K,V>,Object>> filterFunc, Duration _windowDuration, Duration _slideDuration, Partitioner partitioner, scala.reflect.ClassTag<K> evidence$1, scala.reflect.ClassTag<V> evidence$2)

Method Summary

Methods

Modifier and Type	Method and Description
DStream<scala.Tuple2<K,V>>	checkpoint(Duration interval) Enable periodic checkpointing of RDDs of this DStream
scala.Option<RDD<scala.Tuple2<K,V>>>	compute(Time validTime) Method that generates a RDD for the given time
scala.collection.immutable.List<DStream<scala.Tuple2<K,V>>>	dependencies()
boolean	mustCheckpoint()
Duration	parentRememberDuration()
DStream<scala.Tuple2<K,V>>	persist(StorageLevel storageLevel) Persist the RDDs of this DStream with the given storage level
DStream<scala.Tuple2<K,V>>	reducedStream()
Duration	slideDuration() Time interval after which the DStream generates a RDD
Duration	windowDuration()

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpointData, checkpointDuration, clearCheckpointData, clearMetadata, context, count, countByValue, countByValueAndWindow, countByWindow, creationSite, filter, flatMap, foreach, foreach, foreachRDD, foreachRDD, generatedRDDs, generateJob, getCreationSite, getOrCompute, glom, graph, initialize, isInitialized, isTimeValid, map, mapPartitions, persist, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsTextFiles, setContext, setGraph, slice, slice, ssc, storageLevel, transform, transform, transformWith, transformWith, union, updateCheckpointData, validate, window, window, zeroTime

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

ReducedWindowedDStream

public ReducedWindowedDStream(DStream<scala.Tuple2<K,V>> parent,
                      scala.Function2<V,V,V> reduceFunc,
                      scala.Function2<V,V,V> invReduceFunc,
                      scala.Option<scala.Function1<scala.Tuple2<K,V>,Object>> filterFunc,
                      Duration _windowDuration,
                      Duration _slideDuration,
                      Partitioner partitioner,
                      scala.reflect.ClassTag<K> evidence$1,
                      scala.reflect.ClassTag<V> evidence$2)

Method Detail

reducedStream

public DStream<scala.Tuple2<K,V>> reducedStream()

windowDuration

public Duration windowDuration()

dependencies

public scala.collection.immutable.List<DStream<scala.Tuple2<K,V>>> dependencies()

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates a RDD

Specified by:

slideDuration in class DStream<scala.Tuple2<K,V>>

mustCheckpoint

public boolean mustCheckpoint()

Overrides:

mustCheckpoint in class DStream<scala.Tuple2<K,V>>

parentRememberDuration

public Duration parentRememberDuration()

Overrides:

parentRememberDuration in class DStream<scala.Tuple2<K,V>>

persist

public DStream<scala.Tuple2<K,V>> persist(StorageLevel storageLevel)

Description copied from class: DStream

Persist the RDDs of this DStream with the given storage level

Overrides:

persist in class DStream<scala.Tuple2<K,V>>

checkpoint

public DStream<scala.Tuple2<K,V>> checkpoint(Duration interval)

Description copied from class: DStream

Enable periodic checkpointing of RDDs of this DStream

Overrides:

checkpoint in class DStream<scala.Tuple2<K,V>>

Parameters:

interval - Time interval after which generated RDD will be checkpointed

compute

public scala.Option<RDD<scala.Tuple2<K,V>>> compute(Time validTime)

Description copied from class: DStream

Method that generates a RDD for the given time

Specified by:

compute in class DStream<scala.Tuple2<K,V>>