I got this working, but the actual problem was a floating point error. There may have been other issues, but I have since resolved them if you would like the asset you can find it here: https://assetstore.unity.com/packages/vfx/shaders/sprite-renderer-palette-swap-shader-urp-316189.

Also as buttermilch said I may have been passing incorrectly using vertex color, although I think it's used here for the purposes of lighting since this is a lit sprite shader, idk this is like 75% copy pasted from the default unity sprite lit shader.