I managed to make it work by explicitly installing SSH in the vagrant box. I found out while trying a docker instance of Alma9 where I had a lot of trouble with ssh. I don't quite understand why it happens, but seems to be a matter of security restrictions in latest versions of Alma9, so now when it would create the machine with some basic packages, you may need to explicitly install them.

alma9.vm.provision "setup", type: "shell", preserve_order: true, inline: <<-EOF
      echo "Install OpenSSH"
      yum -y install openssh
EOF