79292964

Date: 2024-12-19 02:25:31
Score: 0.5
Natty:
Report link

Streaming tables inherit the processing guarantees of Apache Spark Structured Streaming and are configured to process queries from append-only data sources, where new rows are always inserted into the source table rather than modified.

Once any record is modified in your 'bronze_account_latest', 'bronze_user_latest', these tables cannot be used as streaming source for silver_customer_scd unless you ok to do a 'Full Refresh' on 'silver_customer_scd'or 'skipChangeCommits' as mentioned here https://docs.databricks.com/en/structured-streaming/delta-lake.html#ignore-changes

Alternate is to make 'silver_customer_scd' a materialized view, DLT takes care of refreshing it when underlying tables are updated https://docs.databricks.com/en/views/materialized-views-how-it-works.html

Reasons:
  • Long answer (-0.5):
  • No code block (0.5):
  • Low reputation (0.5):
Posted by: rupaj