Solved by creating a multi-stage Dockerfile which uses the initial stage as the builder and copies over the files from that stage.