Storing the intermediate image in 8-bit or even 16-bit per channel will make you lose way too much precision, maybe 32-bit float will allow you to get a meaningful amount of high frequency detail back, but even that's definitely far from perfect if my math is right.