I experienced similar issues and to speed up TestFlight, I did the following:

1. For missing compliance, I added ITSAppUsesNonExemptEncryption to the info.plist file so I didn't have to seek approval for every build.

2. When uploading new versions, I selected the "TestFlight Only" option under the Select Distribution Method. This not only made TestFlight builds available faster but also prevented me from accidentally submitting the test version for approval.

3. I expired old builds, which sped up the launching of the TestFlight app.

4. I created an internal group and automatically assigned the new version to it so that TestFlight would deliver the build to this group first.