As @david-maze said (thanks!), you need to add steps to the Dockerfile to build in a folder other than the root directory. Add lines like this to your Dockerfile:

COPY cargo.toml build.rs proto src /project
WORKDIR /project

And then you could just add RUN (or not just¹):

RUN cargo build

This will prevent Cargo's scope from scanning unnecessary system's files.

¹ RUN with cache and release:

RUN --mount=type=cache,target=/root/.cargo/registry \
    cargo clean; cargo build --release