What you are looking for is sparse checkout.
git-scm.com/docs/git-sparse-checkout – eftshift0
git sparse-checkout set <unwanted-folder>
This resolved my issues since it stops tracking unwanted folders on my checked-out branch but kept them on remote product branch.
git sparse-checkout disable
to disable this configuration on my local environment