79628509

Date: 2025-05-19 10:05:38
Score: 1.5
Natty:
Report link

The optimize command for a delta table will, by default, keep the old smaller files and just create one or more compacted files + a new version of the delta table in the logs.
It will basically duplicate the data into the new compacted files.

So, if you had 15 files before, you will probably have 16+ now.

This is to keep the possibility to time-travel to the old (non-optimized) version of the table.
See here the Delta Table optimizations docs

As correctly stated by @Veikko, if you want to reduce the amount files (and storage footprint) you need to vacuum the old files.

Reasons:
  • Long answer (-0.5):
  • No code block (0.5):
  • User mentioned (1): @Veikko
  • Low reputation (0.5):
Posted by: AxA