Put the files into a submodule and don't --recurse-submodules on deploy.
For safety, you could disable access to this submodule for the deploy user.

However, I think you may be better off using some CI/CD to create release packages for you to deploy (or have automatically deploy) to your page.