Will need some code snippets to dig further...
There are some similar answers including:
Questions to ask:
- Are they all in the same VPC?
- Is there any difference between scripts?
- Is there any data skew (where some files are much larger than others)
- If you're doing some processing on input files, and the input files are all 200Mb but the transformations create new data, those transforms might create skew in final output but idk
- Are you sure they're all on-demand and not being dropped as spot instances?
- Lastly are you using long term connections throughout? Like do you have the following below:
cnxn = boto3.connect()
process_data_for_a_while()
cnxn.upload(file)
If you do then maybe the cnxn
is too long lived