The issue was that I only had the GPS permissions set to "while using the app" instead of all the time. So it would just hang indefinitely if I ever tabbed over to another app or had the screen off.