79103886

Date: 2024-10-18 23:01:54
Score: 1
Natty:
Report link

so to provide a current data snapshot Hudi has to lookup all commits to obtain locations for all data files, is it right?

not really. the file location are present in the metatata table. the snapshot queries will use the timeline in order to identify valid commits (also exclude invalidated). hudi does not need to store all files in a given snapshot thanks to the ordered timeline design.

note that timeline is archived (keep say 200 commits max), then snapshot queries are not possible before the last non archived commit. btw all files written before are then considered to be read in the sbapshot queries.

last concept: data is stored in file groups. only one commited parquet file of each file group is considered during queries.

Reasons:
  • Blacklisted phrase (0.5): thanks
  • Long answer (-0.5):
  • No code block (0.5):
  • Contains question mark (0.5):
Posted by: parisni