The issue was due to a routing asymmetry in our infrastructure. I went a bit quickly and we could actually not see the ack-syn-synack on the server, only on my machine. So it was discarded on the way back.