Attaching a global key to the local widget surrounding GestureDetector, and using that key to map and fetch point offset solved the issue.