Why not simply scope the pods namespace smaller and include just the pod and needed secret into that namespace. then use (Cluster)Role & RoleBinding limited to that namespace allowing get on secrets.

Your pod then has just access to that secret and not others.