I ended up building a tool to automate this workflow in CI.
It parses ELF + DWARF (per-symbol / per-section / per-file breakdown), stores history across commits, and shows diffs so you can see exactly what grew, where, and by how much. You can also set memory budgets so CI fails when a PR exceeds limits.
The memory report generator is open source:
https://github.com/membrowse/membrowse-action