The problem is incompatibility between LVGL, LVGL driver and ESP-IDF version I have.

I had ESP-IDF v5.3.2 and seems like the basic code of LVGL driver is compatible with v4.*

For that, I had to port part of the code to match the new version of esp-idf.

That solved my issue above, however, still failing on different sizes issue :)