pyspark.sql.streaming.
DataStreamReader
Interface used to load a streaming DataFrame from external storage systems (e.g. file systems, key-value stores, etc). Use SparkSession.readStream to access this.
DataFrame
SparkSession.readStream
Note
Evolving.
New in version 2.0.
__init__
Initialize self. See help(type(self)) for accurate signature.
Methods
__init__(spark)
Initialize self.
csv(path[, schema, sep, encoding, quote, …])
csv
Loads a CSV file stream and returns the result as a DataFrame.
format(source)
format
Specifies the input data source format.
json(path[, schema, primitivesAsString, …])
json
Loads a JSON file stream and returns the results as a DataFrame.
load([path, format, schema])
load
Loads a data stream from a data source and returns it as a DataFrame.
option(key, value)
option
Adds an input option for the underlying data source.
options(**options)
options
Adds input options for the underlying data source.
orc(path[, mergeSchema, pathGlobFilter, …])
orc
Loads a ORC file stream, returning the result as a DataFrame.
parquet(path[, mergeSchema, pathGlobFilter, …])
parquet
Loads a Parquet file stream, returning the result as a DataFrame.
schema(schema)
schema
Specifies the input schema.
text(path[, wholetext, lineSep, …])
text
Loads a text file stream and returns a DataFrame whose schema starts with a string column named “value”, and followed by partitioned columns if there are any.