pyspark.sql.streaming.
StreamingQuery
A handle to a query that is executing continuously in the background as new data arrives. All these methods are thread-safe.
Note
Evolving
New in version 2.0.
__init__
Initialize self. See help(type(self)) for accurate signature.
Methods
__init__(jsq)
Initialize self.
awaitTermination([timeout])
awaitTermination
Waits for the termination of this query, either by query.stop() or by an exception.
query.stop()
exception()
exception
the StreamingQueryException if the query was terminated by an exception, or None.
explain([extended])
explain
Prints the (logical and physical) plans to the console for debugging purpose.
processAllAvailable()
processAllAvailable
Blocks until all available data in the source has been processed and committed to the sink.
stop()
stop
Stop this streaming query.
Attributes
id
Returns the unique id of this query that persists across restarts from checkpoint data.
isActive
Whether this streaming query is currently active or not.
lastProgress
Returns the most recent StreamingQueryProgress update of this streaming query or None if there were no progress updates
StreamingQueryProgress
name
Returns the user-specified name of the query, or null if not specified.
recentProgress
Returns an array of the most recent [[StreamingQueryProgress]] updates for this query.
runId
Returns the unique id of this query that does not persist across restarts.
status
Returns the current status of the query.