I got what the problem was!

1. When I upload an image it has the original size: Image Width: 4032 Image Height: 3024
2. Based on that size Google Vision returns coordinates.
3. But on display Preview image/canvas has a smaller size based on the screen: Preview Image: 391 293
4. So when I tried to draw a polygon based on Original size coordinates it simply could not fit those coordinates into the Preview image size.

Conclusion: Check the image size!