79305610

Date: 2024-12-24 12:37:09
Score: 1.5
Natty:
Report link

You can simply install scrapy-contrib-bigexporter to directly save to parquet from Scrapy: https://github.com/ZuInnoTe/scrapy-contrib-bigexporters

It supports parquet, orc and avro. It has a couple of configuration options that allow you to be flexible (e.g. automatic inference of schema, compression etc.).

See here also an example on how to use parquet: https://github.com/ZuInnoTe/scrapy-contrib-bigexporters/tree/main/examples/quotes_parquet

Reasons:
  • Low length (0.5):
  • No code block (0.5):
  • Low reputation (0.5):
Posted by: Jörn Franke