I encountered the same issue. When I choose to visualize the activations in layer4 of ResNet50, the heatmap is almost identical. However, when I visualize layer3 or layer2, the target location is clearly highlighted.

Could this indicate overfitting in the final layer? In other words, it seems that the earlier layers have already learned enough features, so the final layer doesn't need to do much learning—just passing the features through is sufficient for accurate prediction. Alternatively, it’s possible that the final layer is learning more global features, while the earlier layers focus on more local features, which might explain why the target location is less precise in the heatmap.