I won't be able to explain exactly why, using this config yaml made everything start and work like it was supposed to:
bpf:
hostLegacyRouting: false
cluster:
name: kubernetes
cni:
customConf: false
uninstall: false
ipam:
mode: cluster-pool
operator:
clusterPoolIPv4PodCIDRList:
- 10.244.0.0/16
operator:
replicas: 1
unmanagedPodWatcher:
restart: true
policyEnforcementMode: default
routingMode: tunnel
tunnelPort: 8473
tunnelProtocol: vxlan
If someone knows why this fixed my issue please do still let me know.