In case someone else runs into this issue:

There is a setting on your Twilio account (which seems to default to requiring authentication), which you can disable. It's under Voice > Settings > General