I was also not able to get 'go mod download' working with the ssh option of docker. What you could do instead is save the private key to the ssh folder, execute 'go mod download' and remove the private key. Doing this in a single command ensures that the private key is not saved in one of the layers. Here is a blog post describing how to do that.

https://medium.com/@lightsoffire/how-to-use-golang-private-modules-with-docker-553ff43fa117