79411679

Date: 2025-02-04 12:28:27
Score: 1
Natty:
Report link

Use pandas to read the json file and you can chunksize param in the pd.read_json(). This was you are only loading the few records (in chunks) into memory.

import pandas as pd
    
# Read the JSON file in chunks
for chunk in pd.read_json(input_file, chunksize=chunk_size, lines=True):
    chunk_results = await process_chunk(chunk)

enter image description here

Link to the doc: read_json doc

Reasons:
  • Probably link only (1):
  • Has code block (-0.5):
  • Low reputation (0.5):
Posted by: mohammed_ayaz