Class

org.apache.spark.streaming.kinesis.KinesisInputDStream

Builder

Related Doc: package KinesisInputDStream

Permalink

class Builder extends AnyRef

Builder for KinesisInputDStream instances.

Annotations
@Evolving()
Source
KinesisInputDStream.scala
Since

2.2.0

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Builder()

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  5. def build(): KinesisInputDStream[Array[Byte]]

    Permalink

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    Create a new instance of KinesisInputDStream with configured parameters and using the default message handler, which returns Array[Byte].

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  6. def buildWithMessageHandler[T](handler: (Record) ⇒ T)(implicit arg0: ClassTag[T]): KinesisInputDStream[T]

    Permalink

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    Create a new instance of KinesisInputDStream with configured parameters and the provided message handler.

    handler

    Function converting Record instances read by the KCL to DStream type T

    returns

    Instance of KinesisInputDStream constructed with configured parameters

  7. def checkpointAppName(appName: String): Builder

    Permalink

    Sets the KCL application name to use when checkpointing state to DynamoDB.

    Sets the KCL application name to use when checkpointing state to DynamoDB. This is a required parameter.

    appName

    Value to use for the KCL app name (used when creating the DynamoDB checkpoint table and when writing metrics to CloudWatch)

    returns

    Reference to this KinesisInputDStream.Builder

  8. def checkpointInterval(interval: Duration): Builder

    Permalink

    Sets how often the KCL application state is checkpointed to DynamoDB.

    Sets how often the KCL application state is checkpointed to DynamoDB. Defaults to the Spark Streaming batch interval if no custom value is specified.

    interval

    Duration specifying how often the KCL state should be checkpointed to DynamoDB.

    returns

    Reference to this KinesisInputDStream.Builder

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. def cloudWatchCredentials(credentials: SparkAWSCredentials): Builder

    Permalink

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS CloudWatch endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for CloudWatch authentication

  11. def dynamoDBCredentials(credentials: SparkAWSCredentials): Builder

    Permalink

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS DynamoDB endpoint. Will use the same credentials used for AWS Kinesis if no custom value is set.

    credentials

    SparkAWSCredentials to use for DynamoDB authentication

  12. def endpointUrl(url: String): Builder

    Permalink

    Sets the AWS Kinesis endpoint URL.

    Sets the AWS Kinesis endpoint URL. Defaults to "https://kinesis.us-east-1.amazonaws.com" if no custom value is specified

    url

    Kinesis endpoint URL to use

    returns

    Reference to this KinesisInputDStream.Builder

  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  15. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  16. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  17. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  18. def initialPosition(initialPosition: KinesisInitialPosition): Builder

    Permalink

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to KinesisInitialPositions.Latest if no custom value is specified.

    initialPosition

    KinesisInitialPosition value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

  19. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  20. def kinesisCredentials(credentials: SparkAWSCredentials): Builder

    Permalink

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint.

    Sets the SparkAWSCredentials to use for authenticating to the AWS Kinesis endpoint. Defaults to DefaultCredentialsProvider if no custom value is specified.

    credentials

    SparkAWSCredentials to use for Kinesis authentication

  21. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  22. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  23. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  24. def regionName(regionName: String): Builder

    Permalink

    Sets the AWS region to construct clients for.

    Sets the AWS region to construct clients for. Defaults to "us-east-1" if no custom value is specified.

    regionName

    Name of AWS region to use (e.g. "us-west-2")

    returns

    Reference to this KinesisInputDStream.Builder

  25. def storageLevel(storageLevel: StorageLevel): Builder

    Permalink

    Sets the storage level of the blocks for the DStream created.

    Sets the storage level of the blocks for the DStream created. Defaults to StorageLevel.MEMORY_AND_DISK_2 if no custom value is specified.

    storageLevel

    StorageLevel to use for the DStream data blocks

    returns

    Reference to this KinesisInputDStream.Builder

  26. def streamName(streamName: String): Builder

    Permalink

    Sets the name of the Kinesis stream that the DStream will read from.

    Sets the name of the Kinesis stream that the DStream will read from. This is a required parameter.

    streamName

    Name of Kinesis stream that the DStream will read from

    returns

    Reference to this KinesisInputDStream.Builder

  27. def streamingContext(jssc: JavaStreamingContext): Builder

    Permalink

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    jssc

    JavaStreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

  28. def streamingContext(ssc: StreamingContext): Builder

    Permalink

    Sets the StreamingContext that will be used to construct the Kinesis DStream.

    Sets the StreamingContext that will be used to construct the Kinesis DStream. This is a required parameter.

    ssc

    StreamingContext used to construct Kinesis DStreams

    returns

    Reference to this KinesisInputDStream.Builder

  29. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  30. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  31. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  32. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  33. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def initialPositionInStream(initialPosition: InitialPositionInStream): Builder

    Permalink

    Sets the initial position data is read from in the Kinesis stream.

    Sets the initial position data is read from in the Kinesis stream. Defaults to InitialPositionInStream.LATEST if no custom value is specified. This function would be removed when we deprecate the KinesisUtils.

    initialPosition

    InitialPositionInStream value specifying where Spark Streaming will start reading records in the Kinesis stream from

    returns

    Reference to this KinesisInputDStream.Builder

    Annotations
    @deprecated
    Deprecated

    (Since version 2.3.0) use initialPosition(initialPosition: KinesisInitialPosition)

Inherited from AnyRef

Inherited from Any

Ungrouped