For anyone looking to get the udf's to register and still work with the scala spark session you need to instantiate the python sparksession with a reference to the SparkSession from the gateway. Check the api docs, theres an optional parameter for it in the constructor.