The answer I took from this video: https://youtu.be/mN4259vL4QE
What you need to do is reference EF Core, EF Core Design and any EF Core related provider in infrastructure layer, but you need to change PrivateAssets
property in Microsoft.EntityFrameworkCore.Design
to none or comment whole line.
As you expected there's no need to reference design on startup project, but it will be anyway referenced implicitly, because you have reference to infrastructure layer