79576096

Date: 2025-04-15 22:16:38
Score: 0.5
Natty:
Report link

Spark does not support creating a persistent view using a DataFrame, this is not limited to PySpark.

This is because a persistent view is backed by a "view text", essentially a cleaned SQL query. DataFrames, OTOH, only produces an in-memory query plan without a SQL query in text format, and therefore cannot back a persistent view.

With Spark Connect, now we have a stable API to represent query plans. It would be an interesting project to support this feature by persisting the serialized query plan instead of SQL query text.

Reasons:
  • Long answer (-0.5):
  • No code block (0.5):
  • Low reputation (0.5):
Posted by: Cheng Lian