Just mounting the .ssh folder as explained by @Botje works, the only consideration is that the current user in the container should have the same UID:GID as the owner of the .ssh folder.

My container was running a root user, and my uid/gid was 1000, so I could not use the ssh keys. To solve this, I have created a user in the image with the same uid/gid as mine.