Turns out there is an undocumented option clear_route_cache which solves this issue.
clear_route_cache
https://github.com/envoyproxy/go-control-plane/blob/5d9168428a5cdac76212030e15f47093b0c45a8a/envoy/extensions/filters/http/jwt_authn/v3/config.pb.go#L306