You need to have "Visual Studio Tools for Unity" installed, and then use "Attach to Unity" (or "Attach Unity Debugger") menu option. Native or Managed debugging will not work (as you discovered).

You'll need to have a debug enabled version of the Unity Player. There are some older instructions at https://github.com/dnSpy/dnSpy/wiki/Debugging-Unity-Games#turning-a-release-build-into-a-debug-build on how to do this. https://docs.voxeltycoon.xyz/guides/script-mods/debugging/attaching-a-debugger/#put-the-game-itself-into-development-mode has some newer instructions.

If you want to debug and place breakpoints in the game dll (of which you probably don't have debug pdb's), then https://wiki.fireundubh.com/unity/turning-a-release-build-into-a-debug-build has some tips and tricks on how to do that too.