With @Rukmini's help I was able to figure out that most users in our org got the emails because they were in a group which was in turn assigned the Application Administrator role.

There doesn't seem to be a way to turn off the notifications to the members of Application Administrator but I was able to trim the number of people having this role.