How can you assume that an async operation is quick? If it's a network operation, the user might have a slow internet connection and the operation my take time. If the operation is gps one, the user might have low gpa signal and the operation might take time or never happen.
In your example, lets assume that the user is fetching the location. If the user is able to navigate away from the screen while the setState takes place then you will receive an error.
Same thing happens with other cases, for example while widgets are being animated, when passing a callback to a modal that contains the context ( for example, navigation callback) etc.
Always assume the worst scenario when taking decisions like this. Assume low internet connection, low gps signal, etc.