If you only encounter a slow build in Rider, have a try with disabling the ReSharper build (enabled default), to see if it helps.

ReSharper build is a kind of "wrapper" of dotnet build, will try to utilize all CPU resources to accelerate building process.