org.apache.spark.streaming.dstream

Class InputDStream<T>

Object

org.apache.spark.streaming.dstream.DStream<T>

org.apache.spark.streaming.dstream.InputDStream<T>

All Implemented Interfaces:

java.io.Serializable, Logging

Direct Known Subclasses:

ConstantInputDStream, FileInputDStream, QueueInputDStream, ReceiverInputDStream

public abstract class InputDStream<T>
extends DStream<T>

This is the abstract base class for all input streams. This class provides methods start() and stop() which is called by Spark Streaming system to start and stop receiving data. Input streams that can generate RDDs from new data by running a service/thread only on the driver node (that is, without running a receiver on worker nodes), can be implemented by directly inheriting this InputDStream. For example, FileInputDStream, a subclass of InputDStream, monitors a HDFS directory from the driver for new files and generates RDDs with the new files. For implementing input streams that requires running a receiver on the worker nodes, use ReceiverInputDStream as the parent class.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InputDStream(StreamingContext ssc_, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.immutable.List<scala.runtime.Nothing$>	dependencies()
boolean	isTimeValid(Time time) Checks whether the 'time' is valid wrt slideDuration for generating RDD.
Time	lastValidTime()
Duration	slideDuration() Time interval after which the DStream generates a RDD
abstract void	start() Method called to start receiving data.
abstract void	stop() Method called to stop receiving data.

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, checkpointData, checkpointDuration, clearCheckpointData, clearMetadata, compute, context, count, countByValue, countByValueAndWindow, countByWindow, creationSite, filter, flatMap, foreach, foreach, foreachRDD, foreachRDD, generatedRDDs, generateJob, getCreationSite, getOrCompute, glom, graph, initialize, isInitialized, map, mapPartitions, mustCheckpoint, parentRememberDuration, persist, persist, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsTextFiles, setContext, setGraph, slice, slice, ssc, storageLevel, transform, transform, transformWith, transformWith, union, updateCheckpointData, validate, window, window, zeroTime

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

InputDStream

public InputDStream(StreamingContext ssc_,
            scala.reflect.ClassTag<T> evidence$1)

Method Detail

lastValidTime

public Time lastValidTime()

isTimeValid

public boolean isTimeValid(Time time)

Checks whether the 'time' is valid wrt slideDuration for generating RDD. Additionally it also ensures valid times are in strictly increasing order. This ensures that InputDStream.compute() is called strictly on increasing times.

Overrides:

isTimeValid in class DStream<T>

dependencies

public scala.collection.immutable.List<scala.runtime.Nothing$> dependencies()

slideDuration

public Duration slideDuration()

Description copied from class: DStream

Time interval after which the DStream generates a RDD

Specified by:

slideDuration in class DStream<T>

start

public abstract void start()

Method called to start receiving data. Subclasses must implement this method.

stop

public abstract void stop()

Method called to stop receiving data. Subclasses must implement this method.