The issue was likely caused by the back_log variable which was set to low by default. Setting this variable to a higher value completely resolved the issue. So pity that MariaDB logs don't give a clue of it.
back_log
https://mariadb.com/kb/en/server-system-variables/#back_log