Thank you to Jon Spring for his linked answer:

set position / alignment of ggplot in separate pdf-images after setting absolute panel size

https://patchwork.data-imaginist.com/articles/guides/multipage.html

Using patchwork, I was able to come pretty close.

With patchwork, you can align to plots completely, making their plotting area the same size, no matter their surroundings (legends, axis text, etc.).

By continuing the example from above