79079079

Date: 2024-10-11 16:38:52
Score: 1
Natty:
Report link

Jsonschema is written in Python thus could be slow. I recommend to use Pydantic v2 that is written in Rust. If you have JSON-schema in JSON format you could convert it to Pydantic model with datamodel-code-gen tool:

https://docs.pydantic.dev/latest/integrations/datamodel_code_generator/#code-generation-with-datamodel-code-generator

This solution could be more efficient, but I will be gratefull if someone show me the native pyspark solution to validate complex JSON like could libraries which work on JSON-schema.

Several solutions but None are perfect. From the fastest with PySpark:

  1. explode each nested column with arrays & structs and check them with Spark SQL - isn't sound good
  2. write Scala UDF with Everit JSON library from Maven: https://semanticinsight.wordpress.com/2020/10/18/databricks-spark-json-validation-governance/ and play with run Scala in PySpark - needs effort but possible
  3. Pydantic on the Rust + Python UDF - the simplest way

Apart from that i checked

  1. https://pandera.readthedocs.io/en/stable/pyspark_sql.html - PySpark is a new API in Pandera and in the future could be nice, but now it cannot handle nested data. Only flat DF.
  2. Fugue tool https://fugue-tutorials.readthedocs.io/ tries to convert your custom DF transformation from PySpark to Pandas vectorized UDF which is much faster that common Python UDF. Still nested data is a blocker.
  3. I found great-expectation tool, but still I am checking this out
Reasons:
  • RegEx Blacklisted phrase (2): I will be gratefull
  • Long answer (-1):
  • Has code block (-0.5):
  • Low reputation (0.5):
Posted by: matt91t