79109865

Date: 2024-10-21 11:42:30
Score: 1
Natty:
Report link

If you set the SPARK_HOME environment variable to C:\Spark\spark-3.5.3-bin-hadoop3 and add the path %SPARK_HOME%\bin to your system's PATH environment variable, you can then run your Python script using spark-submit mypythonfile.py. otherwise you should use ./bin/spark-submit script.py

Note:

The pyspark < script.py approach is no longer supported for running Python applications with Apache Spark since version 2.0. You'll encounter an error message like this:

Running Python applications through 'pyspark' is not supported as of Spark 2.0. Use ./bin/spark-submit

Reasons:
  • Long answer (-0.5):
  • No code block (0.5):
  • Low reputation (1):
Posted by: King