The issue was we were using Auth Code Tokens when we should have been using Client Authentication Tokens.
UPS docs are weird as they say they are removing User/PW authentication but just use a form of it to get a Client Auth token. I would have thought the Auth Code tokens would be the "more secure" choice.