While trying this with the current version...
./spark-submit --master k8s://https://127.0.0.1:44617 --deploy-mode cluster --name spark-connect --class org.apache.spark.sql.connect.SimpleSparkConnectService --conf spark.kubernetes.container.image=spark:our-own-apache-spark-with-connect-kb8 --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-connect_2.12-3.5.4.jar
... the driver does not offer the port 15002 but others:
kubectl describe pod spark-connect-d751899504784eaa-driver
...
Ports: 7078/TCP, 7079/TCP, 4040/TCP
So the above
kubectl port-forward NAME_OF_THE_DRIVER 15002
won't work. Any idea why the port is not opened?
Started with this https://medium.com/@SaphE/deploying-apache-spark-on-a-local-kubernetes-cluster-a-comprehensive-guide-d4a59c6b1204 and included the spark connect jar from here https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.4/ since it was not included in the preset docker image.