But it's hard to remove a library, because I can't tell whether a library listed in environment.yml was added by explicit installation or whether it was added as a dependency for a different package that I installed.

This is addressed in Hami's link (https://stackoverflow.com/a/26101973/633093). If you are using the latest conda with mamba solver, conda repoquery whoneeds -t packagename can be used to identify the exact dependencies.

How do I specify project dependencies separate from Conda environments?

To address the main question, environment.yml is still widely used today for maintaining conda environments (not automated but conda tooling has improved a lot). For maintaining project dependencies, poetry is an option: see this related question: Does it make sense to use Conda + Poetry?