This issue was related to new Databricks feature - executor broadcast join https://kb.databricks.com/python/job-fails-with-not-enough-memory-to-build-the-hash-map-error, so to overcome it needed to disable executor broadcast.

Using Databricks notebook autocomplete we found class which contains all databricks-related configurations - com.databricks.sql.DatabricksSQLConf. Inspecting this file public members we found setting which disables executor broadcast join - spark.databricks.execution.executorSideBroadcast.enabled. Disable of executor broadcast resolved our problem - no problem with broadcasting anymore and AQE works fine.

It is too bad that Databricks has a lot of properties which affect query execution, but they are not documented.