pyspark.sql.functions.map_zip_with¶

pyspark.sql.functions.map_zip_with(col1, col2, f)[source]¶

Merge two given maps, key-wise into a single map using a function.

Parameters

col1 – name of the first column or expression
col2 – name of the second column or expression
f – a ternary function (k: Column, v1: Column, v2: Column) -> Column... Can use methods of pyspark.sql.Column, functions defined in pyspark.sql.functions and Scala UserDefinedFunctions. Python UserDefinedFunctions are not supported (SPARK-27052).

Returns

a pyspark.sql.Column

>>> df = spark.createDataFrame([
...     (1, {"IT": 24.0, "SALES": 12.00}, {"IT": 2.0, "SALES": 1.4})],
...     ("id", "base", "ratio")
... )
>>> df.select(map_zip_with(
...     "base", "ratio", lambda k, v1, v2: round(v1 * v2, 2)).alias("updated_data")
... ).show(truncate=False)
+---------------------------+
|updated_data               |
+---------------------------+
|[SALES -> 16.8, IT -> 48.0]|
+---------------------------+

New in version 3.1.

pyspark.sql.functions.map_values pyspark.sql.functions.max