When you build your project it doesn't include anything outside of the src folder. I recently gave a not so good solution to this question about static content where I suggested putting the assets in a public dir at the root but as @Brunnerh pointed out: "The files will not be served with caching headers because the file name is always the same (otherwise you could not properly update the files later)", while his answer solved that issue:

You can import the images via glob import and use that mapping to look up the asset URL.

This is all to explain why your app breaks when building. The solution would be to put your assets within the src folder, maybe in src/lib/static that way you can upload to $lib/static and maybe look into the $lib/server/ directory if you want to implement some server logic to protect those files.