I think the requestMatchers have a higher priority than the method annotations.
I also encountered the same problem. I wanted to validate all resources under the /api/** path, but adding @PermitAll at the /api/abc path was ineffective, and even changing to @PreAuthorize("permitAll()") also didn't work.