As @sonle mentioned, this is the expected behavior. I call it the "Most Permissive Group Rule". This means that when a tester belongs to multiple groups, they will automatically receive builds distributed to the most inclusive group they're part of, regardless of individual group settings.
That's why I prefer setting manual distribution for every group so I have a more granular control.
My key recommendations to mitigate this issue would be:
Create all tester groups with manual distribution by default
Use programmatic scripts integrated into CI pipelines to control build distribution
Carefully manage group memberships to minimize overlap
Implement a dynamic distribution system that provides precise control over which builds are sent to specific groups
By using manual distribution and custom scripts, developers can achieve granular control over build access for different tester groups.
I've worked on this article that covers topics like the one you mentioned. Hope you can use it as a reference!