For those who have the same issue. It's the floating keyboard.

When you have multiple text fields with different keyboard types (mine was alphanumeric and numpad), iOS fails when switching from numpad to alphanumeric keyboard.

My solution is just to change all text fields to use the same keyboard type