I think this method of storing variables will always cause you trouble. You should probably be keeping variables that cross "project" (i.e. environment, deployment) boundaries in your inventory.

The trouble with trying to pick out variables from another folder tree is that there is an algorithm to it. It is loading variables to a group, then the host takes those variables from the group construct.

To try to emulate this you would need to loop through all the groups that host had or {{ group_names }} to see if there is a file to load, or if there is a folder, and if there is a folder, then load all the files in that folder, but only if they are .json, .yml, or .yaml .

And again, back to the original point. If the variables in 'deployment' are applicable to 'environment' based on groups, they really should be in your inventory as group variables.