It turned out to be a network issue. The hostname the bootstrap server was resolving to kafka-target-cluster/10.23.52.37:32185 was also present in my source k8s cluster. So it was not actually connecting to the target cluster.
kafka-target-cluster/10.23.52.37:32185