i did something like this with parquet files
source_base_path = "s3://BUCKET/DB/TABLE/" source_file_path = ["s3://BUCKET/DB/TABLE/*"] df=spark.read.option("basePath",source_base_path).parquet(*source_file_path)