The error message is telling you what's wrong:

"failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "driver": executable file not found in $PATH: unknown"

• Failed to create the containerd task

• unable to start container process exec "driver"

• executable file not found in $PATH unknown

The message is telling you that the driver pod's container is trying to run the command "driver" but it can't find the exec file in the container's path.

You mentioned that --deploy-mode cluster is being used. Spark is trying to launch the driver inside a K8s pod using the Docker image.

This error usually happens when the following occurs:

• The image has no valid ENTRYPOINT or CMD

• Spark is missing from the image

Double check the configuration files (i.e YAML files), the entrypoint is correctly set and the Dockerfile is correct with the CMD.

I have found another StackOverflow that looks similar to help resolve the issue, if not, I'd recommend:

• review the Docker logs

• Check the logs on the EKS pod for any information on K8's end:

  $ kubectl logs <pod name> -n <namespace>

Also giving us more information helps us help you, providing any logs from Docker or kubectl will give us more context/root cause of the issue.