I also faced the same issue, I worked it out after a couple of days brain storming. Here is the detailed explanation as to what is the main issue and how to resolve it. Read out my article on this: https://halahal.in/index.php/2024/12/27/prometheus-grafana-eks/