79510309

Date: 2025-03-14 22:08:07
Score: 2
Natty:
Report link

Sounds like you need to look at Hive Partitioning?

Seems like the perfect solution for your problem, e.g. use the date field of your parquet files as the partition key: https://duckdb.org/docs/stable/data/partitioning/hive_partitioning.html#hive-partitioning

Filters on the partition keys are automatically pushed down into the files. This way the system skips reading files that are not necessary to answer a query. For example, consider the following query on the above dataset:

HTH

Reasons:
  • No code block (0.5):
  • Contains question mark (0.5):
  • Low reputation (1):
Posted by: Dan Bolfter