There was nothing wrong with spark itself – the issue was actually due to a bug (SPARK-42613), which was fixed in version 3.5.0. With this fix, execution now behaves as expected. Since I wasn't explicitly setting spark.task.cpus, it defaulted to 1, which in turn limited the number of cores available to SHAP. After setting spark.task.cpus to the number of cores, the execution time per iteration matched the expected performance!