I think the best option is to use the HDF5 format (via pandas.HDFStore). It lets you store large DataFrames and load only the columns or rows you need, without reading the whole file into memory. But if you need even more flexibility or scalability, then it’s probably time to switch to a proper database.