pyspark.sql.streaming.DataStreamReader.option

DataStreamReader.option(key: str, value: OptionalPrimitiveType) → DataStreamReader

Adds an input option for the underlying data source.

New in version 2.0.0.

Notes

This API is evolving.

Examples

>>> spark.readStream.option("x", 1)
<pyspark.sql.streaming.readwriter.DataStreamReader object ...>

The example below specifies ‘rowsPerSecond’ option to Rate source in order to generate 10 rows every second.

>>> import time
>>> q = spark.readStream.format(
...     "rate").option("rowsPerSecond", 10).load().writeStream.format("console").start()
>>> time.sleep(3)
>>> q.stop()

pyspark.sql.streaming.DataStreamReader.load pyspark.sql.streaming.DataStreamReader.options