I believe the issue you're seeing is an interoperability issue caused by differing codepoints. I've ran into something similar trying to connect an OpenSSL 3.5 client to a BCJSSE 1.80 server.

More specifically, Bouncy Castle 1.80 implements draft-connolly-tls-mlkem-key-agreement-03. The codepoint for specifying ML-KEM-768 in this draft is 0x0768.

On the other hand, OpenSSL 3.5 implements the updated draft-connolly-tls-mlkem-key-agreement-05, which has been replaced by draft-ietf-tls-mlkem. The codepoint for ML-KEM-768 for these drafts is 0x0201. You should be able to validate this with a packet capture.

According to Bouncy Castle release notes, 1.81 should implement the appropriate draft. Upgrading to 1.81 should let your application interoperate with OpenSSL.