I ended up just doing a full docker system prune -a and docker volume prune -a and the issue went away. Not entirely sure what was going on but that worked for me.
docker system prune -a
docker volume prune -a