When looking up the downsides to just replacing all useEffect's with useMemo's, the internet is full of people saying "don't do it!".

UseEffect and useCallback are essentially apples and oranges.

UseEffect does synchronisation. useCallback does code optimisation.

UseEffect yields, extending app's functionality. useCallback yields, improving app's performance.

UseEffect may break because of remounting, in that case the app needs to implement a cleanup function as well. UseCallback has no app specific clean up, instead it just throws-away an old callback.

Answer to the question

Deepening our understanding on useEffect will help us to know how odd to compare it with useCallback. The official documentation is arguably the best on the Internet. May it help you too.