It seems that a workaround for this issue is to make everything else on the page grayscale. I don't know why this is connected to image encoding on the page but it is.
To do grayscale conversion, follow this UniPDF example
https://github.com/unidoc/unipdf-examples/blob/29a15f10d20fa0fbdc256d6382e42812a265494b/advanced/pdf_grayscale_transform.go#L122