I've found the answer and it seems to be pretty straightforward. All I had to do was to listen to the mousedown and mouseup events, those will fire on every click and also return the lat/lng. The same events work on the overlays as well. Also, I now have much more control given that I can distinguish between left and right clicks and what to do between the press and the release of the button.