DataFrameReader.
load
Loads data from a data source and returns it as a DataFrame.
DataFrame
path – optional string or a list of string for file-system backed data sources.
format – optional string for format of the data source. Default to ‘parquet’.
schema – optional pyspark.sql.types.StructType for the input schema or a DDL-formatted string (For example col0 INT, col1 DOUBLE).
pyspark.sql.types.StructType
col0 INT, col1 DOUBLE
options – all other string options
>>> df = spark.read.format("parquet").load('python/test_support/sql/parquet_partitioned', ... opt1=True, opt2=1, opt3='str') >>> df.dtypes [('name', 'string'), ('year', 'int'), ('month', 'int'), ('day', 'int')]
>>> df = spark.read.format('json').load(['python/test_support/sql/people.json', ... 'python/test_support/sql/people1.json']) >>> df.dtypes [('age', 'bigint'), ('aka', 'string'), ('name', 'string')]
New in version 1.4.