After reading the following answer from @SergeyBeryozkin https://github.com/quarkusio/quarkus/discussions/42550 I managed to solve the issue while staying on Quarkus version 3.11.1 as follows:

My properties (note mp not smallrye):

mp.jwt.verify.publickey=<base64EncodedSecret>
mp.jwt.verify.publickey.algorithm=HS512

where <base64EncodedSecret> is the encoding of the complete contents of my original secret.jwk file. Although slightly confusing, as it is not using public keys, it works if upgrading the Quarkus version is not an option.