To answer my own question, I managed this with one queue in the end, just filtering the s3 files with the glob filter