79090106

Date: 2024-10-15 13:17:04
Score: 2.5
Natty:
Report link

A slightly modified version from Herlandro Hermogenes answer (https://stackoverflow.com/a/79067945/27700804)

def download_blob_rate_limited(blob, dest_file, rate_limit=512*1024, freq=10):

    chunk_size = int(rate_limit/freq)
    
    with open(dest_file, 'wb') as file_obj:
        start = 0
        blob_size = blob.size
        while start < blob_size:
            end = min(start + chunk_size, blob_size)
            chunk = blob.download_as_bytes(start=start, end=end)
            file_obj.write(chunk)
            time.sleep(1/freq)
            start = end + 1

I find it easier to specify how many times per second I want to download chunks (and thus on what frequency I want to enforce the rate limit) rather than figuring out the chunk size to achieve that.

Reasons:
  • Blacklisted phrase (1): stackoverflow
  • RegEx Blacklisted phrase (1): I want
  • Long answer (-0.5):
  • Has code block (-0.5):
  • Self-answer (0.5):
  • Low reputation (1):
Posted by: Theo