After long digging into parameters and comparing explain (which were very different), I found out that "semijoin=on" in optimizer_switch was the issue. After turning  "semijoin=off" queries were executed in less than sec.
It seems that, in my case, the semijoin was pushing the IN clause at the beginning of the execution list before all other where/join.... comes after and pushing the large output results to a materialized temporary table which was slowing down.
So it's more my experience than a real question, as I didn't find the answer on internet, but I hope that semijoin did get smarter in newest Mariadb versions.