I found out the issues seems to be because I am using the same column as both the hierarchy and dimension. If I explicitly define another column for the hierarchy, like below. The problem goes away.
The DateHierarchy is defined as:
DateHierarchy := 'Calendar'[Date]