79392059

Date: 2025-01-27 20:12:17
Score: 4.5
Natty:
Report link

I am having the exact same issue since we started with DLT 6 months ago. I was hoping someone else would write a bug report to Databricks, but no luck yet :)

What i successfully did in that regard in my personal tenant (not fully verified in real world use cases): define the column as "not nullable" in the provided schema.

This lead in my small test case to an error on the initial run, but then i could just start the pipeline again (without full refresh!) and on the second run it then worked correctly.

As i said, i still need to verify this properly, but if it works that would be an acceptable workaround in my project (for now). Full refresh is in this project a real risk and usually prevented via: "pipelines.reset.allowed": "false".

Reasons:
  • Blacklisted phrase (1): no luck
  • Long answer (-0.5):
  • No code block (0.5):
  • Me too answer (2.5): I am having the exact same issue
  • Low reputation (1):
Posted by: Thomas