When using Podsubnet feature, src IP is always Pod IP, if it is not going through public network.
This answer DOES NOT ALWAYS apply when not using Podsubnet. In conclusion:

Podsubnet: Pod IP
Nodesubnet: cross VNet = node IP; within VNet = Pod IP
kubenet / Overlay: node IP

Also: AppGw is ingress. No egress.

Is there some way to handle this dynamic behavior?

Since StaticEgressGateway is not an option for you, you may want to check: https://learn.microsoft.com/en-us/azure/aks/http-proxy

But if your application not supporting HTTP_PROXY, you can discard this way.

Or setting UDR + Virtual Appliance (like Azure Firewall), but it is high cost, which I believe is not in your consideration.