I had "agent refused operation" issue with ed25519-sk and ecdsa-sk keys (using yubikey). After setting up with instructions here first ssh login was success but subsequent failed.

Adding IdentityAgent none to ~/.ssh/config or calling ssh command with -o IdentityAgent=none option fixed it. More here.