Posting an answer if anyone has this exact problem - kudos to @Grismar in comments

Setting ssl_verify_client optional_no_ca; will allow the handshake to complete and $ssl_client_verify will be set to FAILED:unable to verify the first certificate which is what I wanted to achieve. It will still work as before when the client has no cert at all (ssl_client_verify is set to NONE)