There is a simpler syntax:
dtgrouped2[dt2, ':='(Amount1 = Amount1,
Amount2 = Amount2),
on = .(Unique = Unique1, MonthNo > StartDate, MonthNo <= EndDate)]
dtgrouped2
MonthNo Unique Total Amount1 Amount2
<int> <char> <int> <int> <int>
1: 1 AAA 10 7 0
2: 1 BBB 0 NA NA
3: 2 CCC 3 NA NA
4: 2 DDD 0 NA NA
5: 3 AAA 0 3 2
6: 3 BBB 35 NA NA
7: 4 CCC 15 NA NA
8: 4 AAA 0 3 2
9: 5 BBB 60 NA NA
10: 5 CCC 0 NA NA
11: 6 DDD 100 NA NA
12: 6 AAA 0 NA NA