79439442

Date: 2025-02-14 12:59:33
Score: 2
Natty:
Report link

While trying this with the current version...

./spark-submit --master k8s://https://127.0.0.1:44617 --deploy-mode cluster --name spark-connect --class org.apache.spark.sql.connect.SimpleSparkConnectService --conf spark.kubernetes.container.image=spark:our-own-apache-spark-with-connect-kb8 --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark local:///opt/spark/examples/jars/spark-connect_2.12-3.5.4.jar

... the driver does not offer the port 15002 but others:

kubectl describe pod spark-connect-d751899504784eaa-driver
...
Ports:         7078/TCP, 7079/TCP, 4040/TCP

So the above

kubectl port-forward NAME_OF_THE_DRIVER 15002

won't work. Any idea why the port is not opened?

Started with this https://medium.com/@SaphE/deploying-apache-spark-on-a-local-kubernetes-cluster-a-comprehensive-guide-d4a59c6b1204 and included the spark connect jar from here https://repo1.maven.org/maven2/org/apache/spark/spark-connect_2.12/3.5.4/ since it was not included in the preset docker image.

Reasons:
  • Blacklisted phrase (0.5): medium.com
  • Probably link only (1):
  • Long answer (-0.5):
  • Has code block (-0.5):
  • Contains question mark (0.5):
  • Low reputation (1):
Posted by: user29644975