Jsonschema is written in Python thus could be slow. I recommend to use Pydantic v2 that is written in Rust. If you have JSON-schema in JSON format you could convert it to Pydantic model with datamodel-code-gen tool:
https://docs.pydantic.dev/latest/integrations/datamodel_code_generator/#code-generation-with-datamodel-code-generator
This solution could be more efficient, but I will be gratefull if someone show me the native pyspark solution to validate complex JSON like could libraries which work on JSON-schema.
Several solutions but None are perfect. From the fastest with PySpark:
- explode each nested column with arrays & structs and check them with Spark SQL - isn't sound good
- write Scala UDF with Everit JSON library from Maven: https://semanticinsight.wordpress.com/2020/10/18/databricks-spark-json-validation-governance/ and play with run Scala in PySpark - needs effort but possible
- Pydantic on the Rust + Python UDF - the simplest way
Apart from that i checked
- https://pandera.readthedocs.io/en/stable/pyspark_sql.html - PySpark is a new API in Pandera and in the future could be nice, but now it cannot handle nested data. Only flat DF.
- Fugue tool https://fugue-tutorials.readthedocs.io/ tries to convert your custom DF transformation from PySpark to Pandas vectorized UDF which is much faster that common Python UDF. Still nested data is a blocker.
- I found
great-expectation tool, but still I am checking this out