So I kind of found the issue and a fix to it.

The problem is that collabora and nextcloud servers being proxied by cloudflare. Specifying the nextcloud-server in aliasgroup1 therefore doesn't allow the actual requesting ip. Meaning: a1.a2.a3.a4 maps to example.com and is listed in aliasgroup1 as the domain of the nextcloud server. The outgoing request from that server comes from b1.b2.b3.b4 which is not mapped to that domain and also not allowed to make WOPI requests, just like any cloudflare ips aren't except the one that maps to example.com.

The following (especially the Collabora allow-list for WOPI requests part) for some reason did solve my problem, maybe because the cloudflare ip also gets parsed? Just adding all Cloudflare IP-Ranges to Allow list for WOPI requests (in Nextcloud instance) fixed the problem for me. I am not sure if this is secure (at all)

https://www.domsky.cz/nextcloud-with-collabora/

Just wanted to leave this here for anyone being confused by the mess that this thread has become over time. @Red's answer is still kind of correct and would work for static ips.