The parent directory structure and and folder naming have specific roles:

Parent Directory 0 acts as the initial namespace for checkpointing when a streaming query is started. The next folder or Parent Directory 1, would be created in scenarios such as restart of the query or state re-initialization.

This ensures that metadata is organized and Spark can recover exactly once semantics, allowing Spark to differentiate between diff runs or phases of the job.