A few years later, but this guide is a great explanation of what's going on:
https://community.databricks.com/t5/technical-blog/deep-dive-streaming-deduplication/ba-p/105062