What I ended up doing was reading the Paket documentation. I realised that it uses the paket.dependencies and paket.lock file to figure out what dependencies it needs to install.

So in the Dockerfile, I first copied in these two files and then did the paket restore before copying in the rest of the sourcecode and building it. This allows these two layers to be cached, and they don't have to be re-run unless the paket.dependencies or paket.lock changes.