pyspark.sql.functions.map_filter

pyspark.sql.functions.map_filter(col, f)[source]

Returns a map whose key-value pairs satisfy a predicate.

Parameters

  • col – name of column or expression

  • f – a binary function (k: Column, v: Column) -> Column... Can use methods of pyspark.sql.Column, functions defined in pyspark.sql.functions and Scala UserDefinedFunctions. Python UserDefinedFunctions are not supported (SPARK-27052).

Returns

a pyspark.sql.Column

>>> df = spark.createDataFrame([(1, {"foo": 42.0, "bar": 1.0, "baz": 32.0})], ("id", "data"))
>>> df.select(map_filter(
...     "data", lambda _, v: v > 30.0).alias("data_filtered")
... ).show(truncate=False)
+--------------------------+
|data_filtered             |
+--------------------------+
|[baz -> 32.0, foo -> 42.0]|
+--------------------------+

New in version 3.1.

pyspark.sql.functions.map_entries pyspark.sql.functions.map_from_arrays